A fully end-to-end multi-modal dialogue sentiment recognition method and system

Abstract

This invention discloses a fully end-to-end multimodal dialogue emotion recognition method and system. The method includes the following steps: segmenting a video file into multiple sub-videos based on utterances; extracting video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video; performing cross-modal fusion of the video modal temporal features, audio modal temporal features, and text modal temporal features for each sub-video; performing temporal modeling of the utterances of the target sub-video with the speaker's own historical utterances and the historical utterances of others, and finally fusing the two models as the final temporal modeling feature output and the input to a subsequent linear classifier. This invention fully utilizes multimodal features and considers the speaker's own emotional changes and emotional transfer between speakers, significantly improving the accuracy and robustness of emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition, and more specifically, relates to a fully end-to-end multimodal dialogue emotion recognition method and system. Background Technology

[0002] In the information age, human communication methods are becoming increasingly diverse, with online communication becoming an integral part of daily life. In recent years, advancements in artificial intelligence and the enhanced communication capabilities of smart devices have led to the widespread adoption of intelligent interactive platforms such as intelligent voice assistants, online customer service support systems, and intelligent automotive systems. These advancements have not only brought convenience and intelligence to our daily lives but have also significantly improved work and production efficiency. As a key factor, emotion recognition is gaining increasing attention in both industry and academia for improving human-computer interfaces and the user experience of voice communication. With the diversification of communication methods, people are paying attention not only to the content of speech but also to the speaker's emotional state. Emotion is an indispensable part of human communication, influencing people's thinking, decision-making, and behavior. Enabling machines to understand and gradually master people's emotional changes is a major research direction in the field of artificial intelligence.

[0003] Early research focused primarily on text sentiment analysis, including rule-based and machine learning-based methods. Subsequently, with the rise of deep learning, researchers began exploring sentiment recognition models based on deep neural networks. These models typically utilize recurrent neural networks (RNNs) and convolutional neural networks (CNNs) to process text data and employ long short-term memory networks (LSTMs) and attention mechanisms to improve the accuracy of sentiment recognition. In speech sentiment analysis, researchers extract acoustic features of speech signals, such as pitch, energy, and frequency, to identify the speaker's emotional state.

[0004] Considering the incompleteness of feature information extracted based on a single modality, recent research has begun to consider multimodal dialogue emotion recognition. Multimodal dialogue emotion recognition is a cutting-edge research topic in affective computing. It utilizes information from multiple modalities, including speech, text, and vision, combined with the context of the target utterance, to analyze the emotional state of the target utterance from a dialogue. This method can more comprehensively capture user emotions, improving the accuracy and effectiveness of emotion recognition. Multimodal dialogue emotion recognition has significant research value and broad application prospects in various fields such as dialogue systems, smart classrooms, and autism intervention. However, existing multimodal dialogue emotion recognition technologies suffer from low accuracy. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a fully end-to-end multimodal dialogue emotion recognition method and system. It fully utilizes the features of multimodality and considers the speaker's own emotional changes as well as the emotional transfer between speakers, greatly improving the accuracy and robustness of emotion recognition.

[0006] To achieve the above objectives, according to one aspect of the present invention, a fully end-to-end multimodal dialogue emotion recognition method is provided, comprising the steps of:

[0007] The video file in the dialogue scene is obtained, and the video file is segmented into multiple sub-videos based on the speech. Each sub-video is marked with the speaker identifier corresponding to that sub-video.

[0008] The video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video are extracted separately.

[0009] Cross-modal fusion is performed on the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video;

[0010] Based on the speaker identifier of each sub-video, the attention mechanism is used to obtain the self-historical discourse features by combining the first fusion feature of each sub-video with the speaker's own historical features. The attention mechanism is also used to obtain the other party's historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker. The self-historical discourse features and other party's historical discourse features of each sub-video are then fused to obtain the second fusion feature.

[0011] The emotion recognition result for each sub-video is output based on the second fusion feature of each sub-video.

[0012] Furthermore, the step of extracting the video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video respectively includes the following steps:

[0013] Extract the corresponding audio and text files from each sub-video;

[0014] Extract video features, audio features, and text features from each sub-video and its corresponding audio and text files, respectively;

[0015] Speech-level temporal modeling was performed on the video features, audio features, and text features of each sub-video to obtain the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video.

[0016] Furthermore, the step of extracting video features, audio features, and text features from each sub-video and its corresponding audio and text files respectively includes the following steps:

[0017] The RoBERTa pre-trained model was used to extract text features from the text files corresponding to each sub-video.

[0018] The Wav2Vec2.0 pre-trained model was used to extract audio features from the audio files corresponding to each sub-video.

[0019] The SwinTransformer pre-trained model is used to extract video features from each sub-video.

[0020] Furthermore, TransformerEncoder is used to perform discourse-level temporal modeling on the video features, audio features, and text features of each sub-video.

[0021] Furthermore, the fusion of the video modal temporal features, audio modal temporal features, and text modal temporal features for each sub-video includes the following steps:

[0022] The interaction features between each modality's temporal features and the temporal features of the other two modalities are calculated using an attention mechanism to obtain two interaction features for each modality.

[0023] By fusing two interactive features of each modality through low-rank tensor fusion, cross-modal features of each modality are obtained.

[0024] Cross-modal features of the three modalities are fused by direct concatenation and then dimensionality is reduced using a linear layer.

[0025] Furthermore, the formula for calculating the interaction features between each modal temporal feature and the other two modal temporal features using the attention mechanism is as follows:

[0026]

[0027] in, This represents the interaction features between the text modal temporal features and the audio modal temporal features of the i-th sub-video. This represents the interaction features between the text modal temporal features and the video modal temporal features of the i-th sub-video. This represents the interaction features between the audio modal temporal features and the text modal temporal features of the i-th sub-video. This represents the interaction feature between the audio modal temporal features and the video modal temporal features of the i-th sub-video. This represents the interaction features between the video modal temporal features and the text modal temporal features of the i-th sub-video. The interaction features between the video modal temporal features and the audio modal temporal features of the i-th sub-video are represented by Attention(.), which represents the attention mechanism. Att-f ti Att-f represents the text modal temporal feature of the i-th sub-video. ai Att-f represents the audio modal temporal features of the i-th sub-video. vi This represents the temporal characteristics of the video modality of the i-th sub-video;

[0028] The calculation formula for fusing the two interaction features of each modality using low-rank tensor fusion is as follows:

[0029]

[0030] in, Represents the cross-modal features of the text modality of the i-th sub-video. This represents the cross-modal feature of the audio modality of the i-th sub-video. Let LMF(.) represent the cross-modal features of the video modality of the i-th sub-video, and let LMF(.) represent low-rank tensor fusion.

[0031] Furthermore, the calculation formula for obtaining the self-historical discourse features using the attention mechanism based on the first fusion feature of each sub-video and the speaker's own historical features in that sub-video is as follows:

[0032]

[0033] in, fuse represents the self-historical discourse characteristics of the i-th sub-video. i This represents the first fusion feature of the i-th sub-video. Let represent the speaker's own historical features in the i-th sub-video, and Attention(.) represent the attention mechanism;

[0034] The calculation formula for obtaining the historical discourse features of others by utilizing the attention mechanism based on the first fusion feature of each sub-video and the historical features of others other than the speaker in that sub-video is as follows:

[0035]

[0036] in, This represents the historical discourse features of others in the i-th sub-video. This represents the historical features of others besides the speaker in the i-th sub-video.

[0037] Furthermore, the step of segmenting the video file into multiple sub-videos by discourse, with each sub-video labeled with a speaker identifier, includes the following steps:

[0038] The video file is segmented into multiple sub-videos based on the speech units. The feature vector of each sub-video is then input into the speaker classification auxiliary task module to obtain the speaker identifier corresponding to each sub-video.

[0039] According to another aspect of the present invention, a fully end-to-end multimodal dialogue emotion recognition system is provided, comprising:

[0040] The module for uploading dialogue video files is used to obtain video files from dialogue scenarios.

[0041] The video speech segmentation module is used to segment video files into speech units, resulting in multiple sub-videos. Each sub-video is marked with the speaker identifier corresponding to that sub-video.

[0042] The emotion recognition module includes a feature extraction module, a first feature fusion module, a second feature fusion module, and a prediction module.

[0043] The feature extraction module is used to extract the video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video, respectively.

[0044] The first feature fusion module is used to perform cross-modal fusion of the video modal temporal features, audio modal temporal features and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video;

[0045] The second feature fusion module, based on the speaker identifier of each sub-video, uses an attention mechanism to obtain the self-historical discourse features of each sub-video by combining the first fusion feature of each sub-video with the speaker's own historical features, and uses an attention mechanism to obtain the other party's historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker of that sub-video. The self-historical discourse features and the other party's historical discourse features of each sub-video are then fused to obtain the second fusion feature.

[0046] The prediction module is used to output the sentiment recognition result of each sub-video based on the second fusion feature of each sub-video.

[0047] Overall, compared with the prior art, the technical solutions conceived in this invention fully utilize the features of multimodal thinking and consider the speaker's own emotional changes as well as the emotional transfer between speakers, greatly improving the accuracy and robustness of emotion recognition. Specifically, this is reflected in the following aspects:

[0048] (1) This invention makes full use of the features of three modalities: text, speech and video, and adopts a segmented context modeling approach to perform dialogue temporal modeling. By calculating the emotional relationship weights of the target discourse’s self-historical discourse and the other’s historical discourse respectively, and fusing the calculated relationship features, it extracts two types of dialogue context emotional information, including the emotional changes within the speaker and the emotional transfer between speakers.

[0049] (2) This invention combines the advantages of multiple fusion methods to perform multimodal fusion. First, the interaction pairs of each modality are extracted through the Attention mechanism. Then, the interaction pairs of each modality are fused through LMF. Finally, they are concatted together as the final fusion feature, which fully considers the emotional features within the modality and the emotional association between modalities.

[0050] (3) The present invention designs a speaker classification auxiliary task. During the training process, the model parameters are optimized through the emotion classification task and the speaker auxiliary task. At the same time, the parameters of the fully connected layer of the discourse segmentation network are fine-tuned to help the discourse segmentation network to more accurately identify the speaker's identity, thereby reducing the error in distinguishing between one's own historical discourse and the historical discourse of others when modeling the context sequence.

[0051] (4) The present invention also employs a speaker segmentation network to segment the complete dialogue video, realizing a complete task chain from dialogue video input to sentiment prediction output of each discourse.

[0052] (5) The present invention also uses pre-trained models of each modality to extract modal features, and fine-tunes the pre-trained models during the training process to obtain more effective emotional features. Attached Figure Description

[0053] Figure 1 This is a schematic diagram illustrating the working principle of the fully end-to-end multimodal dialogue emotion recognition method according to an embodiment of the present invention.

[0054] Figure 2 This is a network diagram of the fully end-to-end multimodal dialogue emotion recognition method according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a fully end-to-end multimodal dialogue emotion recognition system according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0057] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. "Multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0058] Unless otherwise stated, "multiple" means two or more.

[0059] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0060] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] This invention provides a fully end-to-end multimodal dialogue emotion recognition method and system, which will be described below.

[0063] The working principle of a fully end-to-end multimodal dialogue emotion recognition method according to an embodiment of the present invention is as follows: Figure 1 As shown.

[0064] An embodiment of the present invention provides a fully end-to-end multimodal dialogue emotion recognition method, comprising steps 1 to 5.

[0065] Step 1: Obtain the video file of the dialogue scene, and perform speech segmentation on the video file by speech to obtain multiple sub-videos. Each sub-video is marked with the speaker identifier corresponding to that sub-video.

[0066] The dialogue video file (Dialogue-Video) and the dialogue audio file (Dialogue-Audio) are input into the speaker segmentation network AVR_Net to obtain speaker speech segmentation information. The speech segmentation information obtained by AVR_Net can then be used to split the dialogue video file into individual speech video files (Uttr-Video), also known as sub-videos. The AVR_Net network can utilize the official API, which uses a pre-trained AVR_Net network with pre-trained parameters. The audio file (Dialogue-Video) can be extracted from the video file (Dialogue-Video).

[0067] The obtained speaker segmentation information includes: the start timestamp (start), end timestamp (end), and speaker identifier (speaker) for each utterance, which can be expressed by the formula:

[0068] (Starts,Ends,Speakers,Fea)=AVR-Net(Dialogue-Audio,Dialogue-Video)

[0069] Where Starts represents the start timestamp of each utterance in the dialogue video, Ends represents the end timestamp of each utterance in the dialogue video, Speakers represents the speaker identifier of each utterance in the dialogue video, and Fea represents the corresponding output feature of each utterance in the dialogue video.

[0070] Furthermore, after segmenting the video file into segments based on utterances, multiple sub-videos, i.e., multiple utterances, are obtained. To help the utterance segmentation network better classify speakers, the output feature vector obtained by the utterance segmentation network can be input into the speaker classification auxiliary task module to obtain the classification prediction output. The speaker classification auxiliary task module consists of two fully connected layers, where the output feature dimension of the second layer represents the number of speakers in the dialogue. If it is set to 2, it indicates a binary dialogue, and the feature vector output corresponding to the i-th utterance is represented by Fea. i This can be expressed by the formula:

[0071] speaker′ i =argmax(softmax(Linear2(Linear1(Feai )))

[0072] Step 2: Extract the video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video.

[0073] Step 2 includes sub-steps 2.1 to 2.3.

[0074] Step 2.1: Extract the corresponding audio file and text file from each sub-video.

[0075] Specifically, the audio files of each spoken video file can be obtained using the open-source tool FFmpeg, and the text content of each audio file can be obtained using the Baidu Speech Recognition Open API. This can be expressed by the formula:

[0076] Uttr-Audio i =FFmpeg(Uttr-Video) i )

[0077] Uttr-Text i =Baidu-Aip(Uttr-Audio) i )

[0078] Among them, Uttr-Video i Uttr-Audio represents the i-th sub-video. i Uttr-Text represents the audio corresponding to the i-th sub-video. i This represents the text corresponding to the i-th sub-video, and Baidu-Aip() indicates that it is accessed through the Baidu Speech Recognition Open API.

[0079] In this way, we have obtained the audio, text, and video files of each utterance from the dialogue video files. Next, we will use the pre-trained models of each modality to extract the corresponding modal feature information of each utterance.

[0080] Step 2.2: Extract video features, audio features, and text features from each sub-video and its corresponding audio and text files, respectively.

[0081] Specifically, for text, a RoBERTa pre-trained model can be used. RoBERTa (Robustlyoptimized BERT approach) is a pre-trained language model proposed by Facebook AI in 2019. The resulting output features are the text features f of the i-th sub-video. ti f ti The shape is (B, S1, D1), where B is the number of samples, S1 is the text length, and D1 is the hidden layer output dimension of the pre-trained model. This can be represented by the following formula:

[0082] f ti =RoBERTa(Uttr-Text i )

[0083] For audio, the Wav2Vec 2.0 pre-trained model can be used. Wav2Vec 2.0 is a speech recognition model developed by Facebook AI Research (FAIR). The core idea of ​​Wav2Vec 2.0 is to learn the representation of the speech signal through a self-supervised learning method. The resulting output features are the audio features f of the i-th sub-video. ai f ai The shape is (B, S2, D2), where B is the number of samples, S2 is the audio length, and D2 is the hidden layer output dimension of the pre-trained model. This is expressed by the following formula:

[0084] f ai =Wav2Vec2(Uttr-Audio i )

[0085] For videos, the SwinTransformer pre-trained model can be used. SwinTransformer is a novel visual Transformer proposed by Microsoft Research. The main idea of ​​SwinTransformer is to improve the traditional Transformer model to adapt it to the characteristics of computer vision tasks. The resulting output features are the video features f of the i-th sub-video. vi f vi The shape is (B, S3, D3), where B is the number of samples, S3 is the number of video frames, and D3 is the output dimension of the hidden layer of the pre-trained model. This is expressed by the following formula:

[0086] f vi =SwinTransformer(Uttr-Video) i )

[0087] Step 2.3: Perform discourse-level temporal modeling on the video features, audio features, and text features of each sub-video to obtain the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video.

[0088] Then, for each modality, speech-level temporal modeling can be performed using a TransformerEncoder, which can be expressed by the following formula:

[0089] Att-f ti =TransformerEncoder(f ti )

[0090] Att-f ai =TransformerEncoder(f ai )

[0091] Att-f vi =TransformerEncoder(f vi )

[0092] Among them, Att-f vi Att-f ai Att-f ti These are the video modal temporal features, audio modal temporal features, and text modal temporal features of the i-th sub-video, respectively.

[0093] It can retrieve the special marker of the starting word of the input. <cls>Corresponding output.

[0094] A core challenge in dialogue sentiment recognition lies in understanding and modeling the temporal relationships and contextual information within a dialogue. The emotional state in a dialogue is influenced not only by the current utterance but also by the history of previous interactions. The same sentence can convey completely different meanings in different contexts; similarly, for a given single utterance, it can be difficult to accurately judge its sentiment without considering its context. Therefore, temporal modeling of dialogue context is crucial for multimodal sentiment recognition. Currently, mainstream methods for temporal modeling of dialogue context include Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), Gated Recurrent Units (GRUs), and Graph Convolutional Networks (GNNs). However, traditional RNNs face limitations in practical applications, especially when processing long sequences of data. In the case of long sequences, RNNs are prone to gradient vanishing or exploding problems during backpropagation, making it difficult for the network to learn long-range dependencies. Furthermore, RNNs typically have a fixed-size hidden state, meaning all information must be compressed into this state, which may lead to information loss. Traditional RNNs are typically unidirectional, capturing only contextual information from beginning to end. However, for emotion recognition tasks, bidirectional context (considering both preceding and following information) is often more important. While this can be addressed using Bi-RNNs, they still face limitations common to other RNNs. LSTM's gating mechanism, while effective, makes the model computationally more complex and time-consuming. Each time step involves complex matrix operations, which can lead to efficiency issues with long sequences and large datasets. Furthermore, although LSTM is designed to handle long-term dependencies, even it may struggle to capture all dependencies in extremely long sequences. Especially in emotion recognition tasks, emotional cues may be distributed throughout the dialogue or text, and LSTM may still fail to fully capture this scattered information. GRU is another improved RNN. GRU's model structure is simpler than LSTM, requiring fewer parameters, thus reducing training and inference time costs. Like LSTM, GRU can also effectively address the vanishing and exploding gradient problems. However, because GRU simplifies the gating mechanism of LSTM, its memory capacity may be inferior to LSTM in some tasks; furthermore, GRU also requires sequential computation and cannot be parallelized. GNN is not good at capturing such linear temporal dependencies because it is primarily used to capture non-serialized structural information. Moreover, dialogues are dynamic, with new statements constantly being added, and the context of the dialogue changing accordingly. GNN is better suited for relatively static graph structures; it may not be flexible enough for dynamically changing sequence data.Meanwhile, GNN updates the feature representation of nodes by aggregating features from neighboring nodes. This mechanism may not be suitable for capturing the emotional dynamics in dialogue, because emotional state is not only affected by direct neighbors (i.e., sentences before and after), but may also be affected by sentences further away. This influence may not be effectively captured in the traditional structure of GNN.

[0095] The Transformer is a neural network model based on a self-attention mechanism that has achieved great success in the field of Natural Language Processing (NLP). It abandons the structures of RNNs and CNNs, relying entirely on self-attention for computation. The Transformer uses self-attention to process sequential data, allowing the model to directly focus on elements at any position in the sequence. This enables it to capture long-range dependencies, making it ideal for comprehensively understanding contextual information in sentiment recognition tasks. Unlike traditional RNNs and GRUs, the Transformer allows data to be processed in parallel within the model, which greatly improves training efficiency, especially when dealing with large-scale datasets. Transformer models can be easily scaled to larger models and more complex tasks. For example, by increasing the number of layers and adjusting the model size, the model's expressive power and performance can be improved. While RNNs and GRUs typically rely on local context, the Transformer, through its self-attention mechanism, can capture global context, which is crucial for understanding the overall meaning and sentiment of text. Its multi-head attention layer is composed of multiple self-attention layers, which are the core of the Transformer model. These self-attention layers primarily consist of Scaled Dot-Product Attention units and a multi-head attention mechanism, allowing it to capture attention scores based on correlation coefficients across multiple dimensions between vectors. Furthermore, the Transformer does not require fixed-length input sequences, enabling it to handle text data of varying lengths more flexibly, which is highly advantageous for diverse texts in sentiment recognition tasks. Based on this, this invention proposes a Self-Listener Relation Attention module based on the Transformer's attention mechanism to model the temporal relationships within the dialogue context, thereby fully extracting the sentiment information from the context.

[0096] Step 3: Perform cross-modal fusion of the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video.

[0097] Specifically, step 3 may include steps 3.1 to 3.3.

[0098] Step 3.1: Calculate the interaction features of each modal temporal feature with the other two modal temporal features through an attention mechanism.

[0099] Cross-modal attention weights are calculated using an attention mechanism to obtain the interaction pairs of each modality with respect to the other two modalities. The cross-modal interaction pairs of the text modality are represented by the following formula:

[0100]

[0101] Cross-modal interaction pairs in speech modalities are represented by the following formula:

[0102]

[0103] Cross-modal interaction pairs in the visual modality are represented by the following formula:

[0104]

[0105] in, This represents the interaction features between the text modal temporal features and the audio modal temporal features of the i-th sub-video. This represents the interaction features between the text modal temporal features and the video modal temporal features of the i-th sub-video. This represents the interaction features between the audio modal temporal features and the text modal temporal features of the i-th sub-video. This represents the interaction feature between the audio modal temporal features and the video modal temporal features of the i-th sub-video. This represents the interaction features between the video modal temporal features and the text modal temporal features of the i-th sub-video. This represents the interaction features between the video modal temporal features and the audio modal temporal features of the i-th sub-video.

[0106] Step 3.2: Fuse the two interaction features of each modality by low-rank tensor fusion to obtain the cross-modal features of each modality.

[0107] This can be expressed by the following formula:

[0108]

[0109] At this point, we have obtained the feature vectors of each modality that contain cross-modal information.

[0110] in, Represents the cross-modal features of the text modality of the i-th sub-video. This represents the cross-modal feature of the audio modality of the i-th sub-video. Let LMF(.) represent the cross-modal features of the video modality of the i-th sub-video, and let LMF(.) represent low-rank tensor fusion.

[0111] Step 3.3: The cross-modal features of the three modalities are fused by direct concatenation and then dimensionality is reduced by a linear layer.

[0112] This can be expressed by the following formula:

[0113]

[0114] Among them, fuse i This represents the first fusion feature of the i-th sub-video.

[0115] Currently, common multimodal fusion methods include direct concatenation, gating mechanism fusion, low-rank tensor fusion, and graph convolution fusion. Direct concatenation is the simplest method, directly concatenating feature vectors from different modalities to form a comprehensive feature representation. It is simple to implement and has low computational cost; it preserves the complete features of each modality without losing information during the fusion process. However, it cannot automatically adjust the importance of features from different modalities, is easily affected by feature dimension imbalance, and ignores the interaction effects between modalities, potentially leading to the loss of information about feature interactions. Gating mechanisms weight features from different modalities, allowing the model to focus on more important information. Through weight allocation, gating mechanisms can provide interpretability for model decisions. This approach adaptively adjusts the weights of different modal features, strengthening important information and suppressing unimportant information. However, when the number of modalities is large, the computational complexity increases, potentially leading to reduced efficiency; it also requires a large amount of data for training to learn effective weight allocation. Low-rank tensor fusion captures interactions between different modalities by constructing a low-rank tensor, aiming to reduce the number of parameters while retaining important interaction information. This approach explicitly models intermodal interactions, capturing richer feature combinations, and due to its low rank, it has relatively fewer parameters and higher computational efficiency. However, tensor decomposition may lose some complex intermodal interaction information and is sensitive to the choice of tensor rank, which needs to be determined experimentally and may require considerable parameter tuning. Graph convolutional networks (GCNs) aggregate and update node features by performing convolution operations on the nodes of a graph and propagating node information within the graph structure. GCN fusion can capture dependencies between nodes and has good generalization ability when processing graph-structured data. It is suitable for situations where there are explicit relationships between modalities. However, current research shows that GCNs are more suitable for data with obvious graph structures and may not be as effective as other fusion methods in some multimodal tasks.

[0116] Compared to cross-modal fusion in existing patents, the cross-modal fusion proposed in this invention extracts interaction pair information of each modality with respect to the other two modalities through an attention mechanism, then fuses the obtained interaction pairs of each modality using the LMF method, and finally concatenates them as the final fusion feature. By combining multiple fusion methods for multimodal fusion, the advantages of each fusion method are fully utilized to effectively extract emotional information within a modality and emotional connections between modalities.

[0117] Step 4: Based on the speaker identifier of each sub-video, use the attention mechanism to obtain the self-historical discourse features by combining the first fusion feature of each sub-video with the speaker's own historical features. Use the attention mechanism to obtain the other-historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker of the sub-video. Combine the self-historical discourse features and the other-historical discourse features of each sub-video to obtain the second fusion feature.

[0118] After obtaining the multimodal fused feature vector, the dialogue context temporal modeling begins. Firstly, the self-historical discourse relation calculation part obtains the self-historical discourse relation weights by applying attention to the self-historical discourse. This can be expressed by the formula:

[0119]

[0120] fuse represents the self-historical discourse characteristics of the i-th sub-video. i This represents the first fusion feature of the i-th sub-video. Let k represent the speaker's own historical characteristics in the i-th sub-video, that is, the first fusion feature of the speaker's history in the i-th sub-video, and k represent the size of the historical discourse window.

[0121] Then, the part that calculates the historical discourse relations of others obtains the weights of the historical discourse relations of others by applying attention to the historical discourse of others. This can be expressed by the following formula:

[0122]

[0123] in, This represents the historical discourse features of others in the i-th sub-video. It represents the historical features of others other than the speaker of the i-th sub-video, that is, the first fusion feature of the history of speakers other than the speaker of the i-th sub-video.

[0124] For example: Suppose there are 5 utterances in a dialogue, with two speakers, A and B. Please provide examples to supplement the specific meaning of your own historical characteristics and those of the speakers.

[0125] Suppose A speaks sentences 1, 3, and 5, and B speaks sentences 2 and 4. For sentence 5, let k be 2, representing the size of the historical discourse window used; then... The first fusion feature of the two most recent historical statements of another person is indicated by the fact that the fifth statement was made by A, so the other person refers to B, which is the first fusion feature of the second and fourth statements. The first fusion feature is the most recent two historical statements that express the self, namely the first and third statements.

[0126] If it is a triadic dialogue, for a triadic dialogue of A, B, and C, and the target discourse is what C said, then the k self-historical discourses are C's most recent k historical discourses, and the k other-historical discourses are the other people's most recent k historical discourses, including A and B.

[0127] Next, the historical discourse relationship information of self and others is fused by direct concatenation to obtain the fusion vector Att-fuse. i This can be expressed by the formula:

[0128]

[0129] Step 5: Based on the second fusion feature Att-fuse of each sub-video i Output the emotion recognition results for each sub-video.

[0130] Finally, we obtain the dialogue context fusion vector Att-fuse. i The data is fed into a linear classifier to obtain the final sentiment classification prediction y′. i It can be expressed by the formula:

[0131]

[0132] Where softmax(.) represents the Softmax function, and argmax(.) represents the argmax function. The parameter matrix represents the linear layer. A linear layer can be represented by a matrix vector and an offset.

[0133] Figure 2 This is a network diagram of a fully end-to-end multimodal dialogue emotion recognition method according to an embodiment of the present invention.

[0134] The training process of the above network is explained in detail below.

[0135] Obtain a training data sample set, which includes dialogue video files, dialogue audio files, and sentiment labels y for each utterance in the dialogue.

[0136] The training data sample set can include two data sample sets, namely the IEMOCAP and MELD public datasets. The IEMOCAP data sample set includes facial expression samples in six categories: happy, neutral, angry, excited, frustrated and sad. The MELD data sample set includes facial expression samples in seven categories: surprised, happy, neutral, angry, disgusted, fearful and sad.

[0137] The sentiment classification prediction y′ is obtained using the methods described in steps 1 to 5 above. i .

[0138] The speaker classification auxiliary task module and the emotion recognition module are trained as a whole. The cross-entropy function is used as the network loss function, where n represents the number of samples, represents the true emotion category, emotion-loss represents the emotion classification loss, and speaker-loss represents the speaker classification loss. Both losses are calculated using the cross-entropy loss function, and the calculation formulas are as follows:

[0139]

[0140] In the formula, speaker i The true label representing the speaker identifier of the training sample, speaker′ i The predicted label y represents the speaker identifier of the training sample. i y′ represents the true label of the speaker identifier in the training sample. i The predicted label represents the speaker identifier of the training sample.

[0141] The overall loss of the model is obtained by weighting the two losses, and can be expressed by the following formula:

[0142] loss=a*emotion-loss+b*speaker-loss

[0143] Furthermore, the value of a is set to 1, and the value of b is set to 0.1;

[0144] The algorithm of this invention achieves a maximum accuracy of 71.94% in sentiment six-class classification on the IEMOCAP training sample dataset and a maximum accuracy of 68.08% in sentiment seven-class classification on the MELD training sample dataset, which is better than most mainstream models.

[0145] This invention discloses a fully end-to-end multimodal dialogue emotion recognition system, comprising a dialogue video file uploading module, a video speech segmentation module, and an emotion recognition module. The emotion recognition module includes a feature extraction module, a first feature fusion module, a second feature fusion module, and a prediction module.

[0146] The video speech segmentation module is used to segment video files into speech units, resulting in multiple sub-videos. Each sub-video is marked with the speaker identifier corresponding to that sub-video.

[0147] The emotion recognition module includes a feature extraction module, a first feature fusion module, a second feature extraction module, and a prediction module.

[0148] The feature extraction module is used to extract the video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video, respectively.

[0149] The first feature fusion module is used to perform cross-modal fusion of the video modal temporal features, audio modal temporal features and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video;

[0150] The second feature fusion module, based on the speaker identifier of each sub-video, uses an attention mechanism to obtain the self-historical discourse features of each sub-video by combining the first fusion feature of each sub-video with the speaker's own historical features, and uses an attention mechanism to obtain the other party's historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker of that sub-video. The self-historical discourse features and the other party's historical discourse features of each sub-video are then fused to obtain the second fusion feature.

[0151] The prediction module is used to output the sentiment recognition result of each sub-video based on the second fusion feature of each sub-video.

[0152] The emotion recognition module can predict the emotion of each utterance in a complete dialogue video file and output an emotion change curve for each utterance. It can also output emotion change curves for different speakers based on their speaker IDs. Specifically, the audio, text, and video file information of each utterance obtained earlier is input into the aforementioned end-to-end multimodal dialogue emotion recognition network to obtain the emotion prediction output for each utterance, and the emotion changes in the dialogue are displayed in the form of a line graph.

[0153] Furthermore, a fully end-to-end multimodal dialogue emotion recognition system may also include a speaker classification auxiliary task module for identifying the speaker identifier corresponding to each sub-video after speech segmentation.

[0154] The fully end-to-end multimodal dialogue emotion recognition system has the same implementation principle and technical effect as the aforementioned fully end-to-end multimodal dialogue emotion recognition method, and will not be repeated here.

[0155] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / cls>

Claims

1. A fully end-to-end multimodal dialogue emotion recognition method, characterized in that, Including the following steps: The video file in the dialogue scene is obtained, and the video file is segmented into multiple sub-videos based on the speech. Each sub-video is marked with the speaker identifier corresponding to that sub-video. The video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video are extracted separately. Cross-modal fusion is performed on the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video; Based on the speaker identifier of each sub-video, the attention mechanism is used to obtain the self-historical discourse features by combining the first fusion feature of each sub-video with the speaker's own historical features. The attention mechanism is also used to obtain the other party's historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker. The self-historical discourse features and other party's historical discourse features of each sub-video are then fused to obtain the second fusion feature. The emotion recognition result for each sub-video is output based on the second fusion feature of each sub-video; The formula for calculating the interaction features between each modality's temporal features and the other two modality temporal features using the attention mechanism is as follows: in, Indicates the first i Interaction features between text modal temporal features and audio modal temporal features of individual sub-videos. Indicates the first i The interaction features between the text modal temporal features and the video modal temporal features of each sub-video. Indicates the first i Interaction features between audio modal temporal features and text modal temporal features of individual sub-videos. Indicates the first i The interaction features between the audio modal temporal features and the video modal temporal features of each sub-video. Indicates the first i The interaction features between the video modal temporal features and the text modal temporal features of each sub-video. Indicates the first i The interaction features between the video modal temporal features and the audio modal temporal features of each sub-video. This represents the attention mechanism. Indicates the first i Textual modal temporal features of individual sub-videos Indicates the first i Audio modal temporal features of each sub-video Indicates the first i Temporal characteristics of video modalities in each sub-video; The calculation formula for fusing the two interaction features of each modality using low-rank tensor fusion is as follows: in, Indicates the first i Cross-modal features of the text modality of each sub-video. Indicates the first i Cross-modal features of audio modalities in individual sub-videos Indicates the first i Cross-modal features of video modalities in individual sub-videos Indicates low-rank tensor fusion; The formula for calculating the self-historical discourse features by utilizing the attention mechanism to obtain the first fusion feature of each sub-video and the speaker's own historical features in that sub-video is as follows: in, Indicates the first i The self-historical discourse characteristics of individual videos Indicates the first i The first fusion feature of each sub-video, Indicates the first i The speaker's own historical characteristics in each video. This represents the attention mechanism; The calculation formula for obtaining the historical discourse features of others by utilizing the attention mechanism based on the first fusion feature of each sub-video and the historical features of others other than the speaker in that sub-video is as follows: in, Indicates the first i The characteristics of others' historical discourse in individual videos. Indicates the first i Historical characteristics of others besides the speaker in the individual video.

2. The fully end-to-end multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The step of extracting video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video includes the following steps: Extract the corresponding audio and text files from each sub-video; Extract video features, audio features, and text features from each sub-video and its corresponding audio and text files, respectively; Speech-level temporal modeling was performed on the video features, audio features, and text features of each sub-video to obtain the video modal temporal features, audio modal temporal features, and text modal temporal features of each sub-video.

3. The fully end-to-end multimodal dialogue emotion recognition method as described in claim 2, characterized in that, The step of extracting video features, audio features, and text features from each sub-video and its corresponding audio and text files includes the following steps: The RoBERTa pre-trained model was used to extract text features from the text files corresponding to each sub-video. The Wav2Vec2.0 pre-trained model was used to extract audio features from the audio files corresponding to each sub-video. The SwinTransformer pre-trained model is used to extract video features from each sub-video.

4. The fully end-to-end multimodal dialogue emotion recognition method as described in claim 2, characterized in that, TransformerEncoder is used to perform discourse-level temporal modeling on the video features, audio features and text features of each sub-video.

5. The fully end-to-end multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The fusion of video modal temporal features, audio modal temporal features, and text modal temporal features for each sub-video includes the following steps: The interaction features between each modality's temporal features and the temporal features of the other two modalities are calculated using an attention mechanism to obtain two interaction features for each modality. By fusing two interactive features of each modality through low-rank tensor fusion, cross-modal features of each modality are obtained. The cross-modal features of the three modalities are fused by direct concatenation and then dimensionality is reduced by a linear layer.

6. The fully end-to-end multimodal dialogue emotion recognition method as described in claim 1, characterized in that, The process of segmenting the video file into multiple sub-videos based on individual utterances, with each sub-video labeled with a corresponding speaker identifier, includes the following steps: The video file is segmented into multiple sub-videos based on the speech units. The feature vector of each sub-video is then input into the speaker classification auxiliary task module to obtain the speaker identifier corresponding to each sub-video.

7. A fully end-to-end multimodal dialogue emotion recognition system, characterized in that, include: The module for uploading dialogue video files is used to obtain video files from dialogue scenarios. The video speech segmentation module is used to segment video files into speech units, resulting in multiple sub-videos. Each sub-video is marked with the speaker identifier corresponding to that sub-video. The emotion recognition module includes a feature extraction module, a first feature fusion module, a second feature fusion module, and a prediction module. The feature extraction module is used to extract the video modal temporal features, audio modal temporal features, and text modal temporal features from each sub-video, respectively. The first feature fusion module is used to perform cross-modal fusion of the video modal temporal features, audio modal temporal features and text modal temporal features of each sub-video to obtain the first fused feature of each sub-video; The second feature fusion module, based on the speaker identifier of each sub-video, uses an attention mechanism to obtain the self-historical discourse features of each sub-video by combining the first fusion feature of each sub-video with the speaker's own historical features, and uses an attention mechanism to obtain the other party's historical discourse features by combining the first fusion feature of each sub-video with the historical features of others other than the speaker of that sub-video. The self-historical discourse features and the other party's historical discourse features of each sub-video are then fused to obtain the second fusion feature. The prediction module is used to output the sentiment recognition result of each sub-video based on the second fusion feature of each sub-video; The formula for calculating the interaction features between each modality's temporal features and the other two modality temporal features using the attention mechanism is as follows: in, Indicates the first i Interaction features between text modal temporal features and audio modal temporal features of individual sub-videos. Indicates the first i The interaction features between the text modal temporal features and the video modal temporal features of each sub-video. Indicates the first i Interaction features between audio modal temporal features and text modal temporal features of individual sub-videos. Indicates the first i The interaction features between the audio modal temporal features and the video modal temporal features of each sub-video. Indicates the first i The interaction features between the video modal temporal features and the text modal temporal features of each sub-video. Indicates the first i The interaction features between the video modal temporal features and the audio modal temporal features of each sub-video. This represents the attention mechanism. Indicates the first i Textual modal temporal features of individual sub-videos Indicates the first i Audio modal temporal features of each sub-video Indicates the first i Temporal characteristics of video modalities in each sub-video; The calculation formula for fusing the two interaction features of each modality using low-rank tensor fusion is as follows: in, Indicates the first i Cross-modal features of the text modality of each sub-video. Indicates the first i Cross-modal features of audio modalities in individual sub-videos Indicates the first i Cross-modal features of video modalities in individual sub-videos Indicates low-rank tensor fusion; The formula for calculating the self-historical discourse features by utilizing the attention mechanism to obtain the first fusion feature of each sub-video and the speaker's own historical features in that sub-video is as follows: in, Indicates the first i The self-historical discourse characteristics of individual videos Indicates the first i The first fusion feature of each sub-video, Indicates the first i The speaker's own historical characteristics in each video. This represents the attention mechanism; The calculation formula for obtaining the historical discourse features of others by utilizing the attention mechanism based on the first fusion feature of each sub-video and the historical features of others other than the speaker in that sub-video is as follows: in, Indicates the first i The characteristics of others' historical discourse in individual videos. Indicates the first i Historical characteristics of others besides the speaker in the individual video.

Images