Multi-modal dialogue emotion recognition system based on relational graph convolutional neural network

Abstract

The invention belongs to the technical field of computer application, and particularly relates to a multi-modal dialogue emotion recognition system based on a relational graph convolutional neural network, and the system comprises a data collection module, a feature determination module and an emotion determination module, the data collection module is used for obtaining video data of a to-be-recognized dialogue, and determining the emotion of the to-be-recognized dialogue; the video data comprises dialogue audio data among a plurality of dialogue parties, the feature determination module determines emotional features corresponding to dialogue statements in the dialogue audio data based on the dialogue audio data, and continuously determines emotional relevance among the dialogue statements based on contexts among the dialogue statements; and the emotion determination module determines an emotion type corresponding to each statement in the to-be-recognized dialogue based on the emotion relevance between the dialogue statements, and improves the accuracy and robustness of emotion recognition through cross-modal multi-feature fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer application, and particularly relates to a multi-modal dialogue emotion recognition system based on a relational graph convolutional neural network BACKGROUND

[0002] With the rapid development of mobile Internet, human-computer interaction has become a common interaction mode in people's daily life. People generally hope that the device (machine) with which they interact has the ability to recognize and understand emotions similar to humans, thereby putting forward higher requirements for human-computer interaction technology.

[0003] In the prior art, GCN network is used to recognize emotions by acquiring multi-modal data. The function of multi-modal dialogue emotion recognition mainly lies in the ability to comprehensively analyze information from different modalities, so as to more accurately recognize and understand the emotional state of users in the dialogue. The multi-modal dialogue emotion recognition system can fuse information from multiple modalities such as text, sound, image, etc. The system can more comprehensively understand the emotional state of the user by comprehensively analyzing these information. However, the technology itself still has certain challenges and limitations: In the process of multi-modal emotion recognition, there are challenges such as large emotional subjectivity, missing of original data, imbalance between modalities, and inaccurate data labels. For example, in some multi-modal emotion recognition applications, the information contained in text, image, speech, and video modalities is not balanced, and different modalities contain different noises. In multi-modal learning, the imbalance between these modalities can easily cause the model to be biased towards a dominant modality. Or in the process of manual data labeling, because each person has different subjective feelings about emotions, the data labels are not accurate, which also brings certain difficulties to model training.

[0004] Therefore, how to more effectively fuse multiple modal information, how to better handle the context information in complex dialogue, and how to improve the robustness of the model on uncertain samples. SUMMARY

[0005] To solve the above problems in the prior art, the application provides a multi-modal dialogue emotion recognition system based on a relational graph convolutional neural network.

[0006] The purpose of the application can be achieved by the following technical solutions: The multi-modal dialogue emotion recognition system based on the relational graph convolutional neural network comprises a data acquisition module, a feature determination module and an emotion determination module which are connected in sequence. The data acquisition module is used to acquire video data of a dialogue to be recognized, and the video data comprises dialogue audio data between a plurality of dialogue parties. The feature determination module determines the emotional features corresponding to each dialogue statement in the dialogue audio data based on the dialogue audio data, and determines the emotional correlation between each dialogue statement based on the context between each dialogue statement. The emotion determination module determines the emotion type corresponding to each statement in the dialogue to be identified based on the emotional correlation between each dialogue statement.

[0007] Preferably, the feature determination module includes a feature extraction unit, a first feature fusion unit, and a second feature fusion unit: The feature extraction unit is used to extract the audio information, image information and text information carried by the video data, determine the text information corresponding to the video data, and use deep learning feature extraction algorithms to extract the audio features, image features and text features from the feature information of the audio information, image information and text information respectively; The first feature fusion unit performs feature fusion on the audio features, image features, and text features using a feature fusion algorithm to obtain a first fused feature; The second feature fusion unit inputs the first fused feature into multiple parallel classifiers, outputs multiple decision results, and then fuses the multiple decision results again to obtain the second fused feature; The emotional correlation between each dialogue statement is determined based on the second fusion feature.

[0008] Preferably, the method by which the second feature fusion unit obtains the second fused feature includes at least one of averaging, voting, weighting, adaptive enhancement, and dynamic Bayesian network.

[0009] Preferably, the emotion determination module includes using a two-layer DRG network to determine the emotion type corresponding to each statement in the dialogue to be identified based on the emotion correlation between each dialogue statement. The relationship graph includes: using the emotion features corresponding to each dialogue statement as graph nodes, and the dialogue relationship type between speakers as directed edges, and connecting each graph node with directed edges. The graph nodes are input into the DRG network, and after convolution operations of multiple layers of convolutional neural networks, the node representations are obtained. Then, a classifier is used to classify the graph nodes and output the emotion type of each dialogue statement.

[0010] Preferably, the emotion determination module further includes an update unit, which extracts the graph structure features of the relationship graph through the feature extraction layer of the DRG network, and updates the nodes according to the relationship type of the directed edges and the information of adjacent discourse nodes.

[0011] Preferably, the update formula of the update unit is: ; in, For the first Nodes after convolution The characteristic representation of R is a set of relation types. For nodes The set of neighboring nodes under relation r The normalization coefficient is... and These are the learnable convolution weight matrices. Used for processing neighbor information. Used to process its own residual information. This is the activation function.

[0012] Preferably, the DRG network includes a normalization operation on the emotional features of each dialogue statement using standard deviation, mean, training scale, and shift vector parameters.

[0013] Preferably, the DRG network uses GraphTransformer to replace the original attention layer, in order to capture and utilize the structural information of the relationship graph and the information between nodes, as well as the structural information and the relationships between nodes.

[0014] Preferably, the feature extraction unit further includes extracting facial expression features from video data containing facial expressions using bi-branch features, the extraction process including: The video data is sliced ​​and frame-sampling to obtain multiple video segments; The process involves performing bi-branch feature extraction on multiple video segments, including: extracting optical flow images from the video segments and using a temporal CNN network to extract dynamic facial expression features from the optical flow images. The speaker's face is cropped from video segments, and a spatial CNN network is used to extract static facial features. The Deep Belief Network (DBN) is used to fuse dynamic facial expression features and static facial features to output facial expression features.

[0015] The beneficial effects of this invention are as follows: The multimodal dialogue emotion recognition model method based on relational graph neural networks adopted in this system clusters nodes into groups after the relational graph neural network, and performs normalization operation on each group of nodes separately to increase the similarity between feature nodes in the same group, and separates the node representations of different groups. Based on the superior performance of the relational graph convolutional neural network multimodal dialogue emotion recognition model on the IEMOCAP dataset four-class classification, it alleviates the problem of confusion in the recognition of similar emotion categories to a certain extent.

[0016] Multimodal dialogue emotion recognition combines multiple data sources such as text, voice, images, and video, and can capture and analyze users' emotional expressions from multiple dimensions. Multimodal dialogue emotion recognition technology can integrate information from different modalities such as text, voice, images, and videos.

[0017] Furthermore, through algorithms such as multimodal joint representation learning in deep learning, data from these modalities are mapped to a unified semantic space. Compared to single emotion recognition (text, speech, image, and video), or combinations of two emotion recognition methods (such as text + speech, text + image, and text + video), this cross-modal fusion achieves information complementarity and enhancement. This cross-modal fusion improves the accuracy and robustness of emotion recognition. Attached Figure Description

[0018] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0019] Fig. 1 This is a system structure block diagram of the present invention; Fig. 2 This is the audio feature extraction process of the feature extraction unit of the present invention; Fig. 3 This is a diagram of the deep learning network framework used by the feature extraction unit of this invention to extract facial expression features. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0021] Multimodal emotion recognition methods refer to using information from two or more of these modalities to build an emotion recognition model. Since the data from different modalities differ significantly in structure and processing, introducing new modal information may bring better performance to the model, but it also increases the complexity and challenge of modeling.

[0022] Methods for multimodal dialogue emotion recognition mainly include those based on Recurrent Neural Networks (RNNs) and Graph Convolutional Neural Networks (GCNs). RNNs suffer from drawbacks such as vanishing and exploding gradients, long training times, and difficulty in acquiring information from distant past. GCNs, however, are not only applicable to text-based dialogue data but can also be extended to other modalities, such as speech and video. This makes GCNs more widely applicable in multimodal dialogue emotion recognition, making them the focus of this research. However, in some multimodal emotion recognition applications, the information content of text, image, speech, and video modalities is unbalanced, and the noise levels differ between modalities. During multimodal learning, this imbalance can easily cause the model to favor one dominant modality. Furthermore, in manual data labeling, individual subjective perceptions of emotion can lead to inaccurate data labels, further complicating model training.

[0023] Based on this, please refer to Figs. 1-3 This embodiment provides a multimodal dialogue emotion recognition system based on a relational graph convolutional neural network, including a data acquisition module, a feature determination module, and an emotion determination module that are sequentially connected in communication. The data acquisition module is used to acquire video data of the dialogue to be recognized. The video data includes audio data of the dialogue between several parties. The feature determination module determines the emotional features corresponding to each dialogue statement in the dialogue audio data based on the dialogue audio data, and determines the emotional correlation between each dialogue statement based on the context between each dialogue statement. The feature determination module includes a feature extraction unit, a first feature fusion unit, and a second feature fusion unit, which are connected in sequence via communication. The feature extraction unit is used to extract audio, image, and text information carried in the video data, determine the text information corresponding to the voice dialogue, and use deep learning feature extraction algorithms to extract audio, image, and text features from the feature information of the audio, image, and text information, specifically including: Part 1: Text Feature Extraction: Deep learning-based feature extraction methods, mainly including Word2vec, BERT, and ELMO, are used to extract text features from audio. Word2vec is essentially a static approach; the word embedding does not change with the context, which has certain limitations. BERT, on the other hand, employs a novel Masked Language Model (MLM), which can generate deep bidirectional language representations. Furthermore, it has strong generalization ability; after pre-training, it only requires adding an extra output layer for fine-tuning to achieve optimal performance on various downstream tasks. It can conveniently achieve good results in text feature extraction. Part Two: Audio Feature Extraction: Audio data can have features extracted using tools such as the librosa library and wav2vec. Commonly used audio features include Mel-Spectrogram and Mel-Frequency Cepstral Coefficients (MFCCs), etc. Please refer to [link to relevant documentation]. Fig. 2 This paper presents an audio feature extraction process. First, the input audio signal undergoes pre-emphasis processing to compensate for high-frequency components and improve high-frequency resolution. Next, frame segmentation is performed, dividing the continuous audio into several short frames to simulate the short-term stationary characteristics of speech signals. Then, each frame is windowed, often using Hamming windows, to reduce signal abrupt changes at the frame ends. The windowed signal is then subjected to Fourier transform to convert the time domain to the frequency domain, obtaining the spectrum. The power spectrum is calculated to highlight energy distribution. An amplitude square operation is performed on the power spectrum. Then, a Mel filter bank is used to simulate human hearing characteristics and extract frequency components related to perception. Finally, two paths are used: one takes the logarithm of the filtering result to obtain the Mel spectrum, and the other takes the logarithm first and then performs a discrete cosine transform to finally output the Mel frequency cepstral coefficients. This completes the extraction from the audio signal to feature representation, providing a foundation for subsequent audio analysis (such as emotion recognition and speech recognition). Part Three: Image Feature Extraction Image feature extraction is a key concept in computer vision and image processing. It refers to using computer algorithms to extract representative information or features from images. A feature is the "interesting" part of a digital image and is the starting point for many computer image analysis algorithms. The definition of a feature is often determined by the problem or application type, and there is no universal or precise definition. One of the most important characteristics of feature extraction is "reproducibility," meaning that the features extracted from different images of the same scene should be the same. Traditional algorithms such as SIFT, SURF, HOG, LBP, and Haar, as well as deep learning techniques such as convolutional neural networks (CNN) and autoencoders, can be used to extract image features such as color, texture, shape, and spatial relationships. Part 4: Video Feature Extraction: Traditional video emotion recognition typically uses manual feature extraction and shallow learning methods, leading to difficulties in dataset extraction. Video emotion recognition primarily identifies emotions through facial expressions, which can be further categorized into static and dynamic emotion recognition based on their features. Static emotion recognition can classify emotions by extracting facial features, generally employing geometric, color, and appearance methods, primarily representing facial shape and muscle expression. Dynamic features, on the other hand, need to consider the relationships between frames. Compared to static image features, dynamic image features take into account more temporal correlation and the continuous changes in facial expressions, with the advantage of reflecting the motion changes of different expressions.

[0024] Facial key points include features such as eyes, nose, mouth, and contours. Since facial key points contain semantic information about the face, they play a significant role in video emotion recognition.

[0025] The first feature fusion unit uses a feature fusion algorithm to fuse audio features, image features, and text features to obtain the first fused feature. The second feature fusion unit inputs the first fusion feature into multiple parallel classifiers, outputs multiple decision results, and then fuses the multiple decision results again to obtain the second fusion feature. The fusion method includes at least one of averaging, voting, weighting, adaptive enhancement, and dynamic Bayesian network. The emotional correlation between each dialogue statement is determined based on the second fusion feature.

[0026] The emotion determination module determines the emotion type of each statement in the dialogue based on the emotional correlation between the statements. Please see Fig. 3 The emotion determination module uses a two-layer DRG network (two-layer relation graph convolutional neural network) to determine the emotion type of each statement in the dialogue based on the emotion correlation between each dialogue statement. The relation graph convolutional neural network (DRG) layer is divided into two parts: the first part constructs the relation graph structure; the second part constructs the DRG.

[0027] Part 1: When constructing a relationship graph, pay attention to the dependencies between nodes. Nodes are utterances in the dialogue process. Dash node is connected by directed edges, and the directed edges represent the dialogue relationship type between speakers.

[0028] Part 2: Construct a single-layer architecture for DRG, use DRG to extract graph structure features, and update nodes based on the relationship type of directed edges and information of neighboring discourse nodes to fully utilize the dependencies between discourses and speakers.

[0029] The relationship graph includes: using the corresponding emotional features in each dialogue statement as graph nodes, using the relationship graph convolutional neural network formula, using a linear model method to cluster the nodes into multiple groups, learning to adjust the training convolution weights, and through the inner product between parameters, nodes with highly similar features are assigned to the same group with high probability. The dialogue relationship between the speakers is a directed edge, and the nodes of the graph are connected by directed edges; The DRG network replaces the original attention layer with a GraphTransformer to capture and utilize the structural information of the graph and the information between nodes, as well as the relationships between these nodes. While traditional Transformer models perform well in processing sequential data, such as text and time series, they are not suitable for directly processing graph-structured data because graph data contains rich structural information and complex node relationships. By introducing graph neural network components, the GraphTransformer can effectively capture and utilize the structural information and relationships between nodes in the graph, thus improving the model's performance on graph structures.

[0030] The DRG network includes normalizing the emotional features of each dialogue utterance using standard deviation, mean, training scale, and shift vector parameters. Nodes in each group are rescaled using specific mean and variance to make nodes in the same group more similar and to separate the node distribution of different groups, preventing nodes in different groups from being too close together. At the same time, residual connections are used to connect splicing nodes with input nodes to prevent normalization from overcorrecting.

[0031] The graph nodes are input into the DRG network, and after convolution operations of multiple layers of convolutional neural networks, the node representations are obtained. Then, a classifier is used to classify the graph nodes and output the emotion type of each dialogue statement. The emotion types include various emotional expressions such as happiness, joy, disgust, and surprise.

[0032] The multimodal dialogue emotion recognition model method based on relational graph neural networks adopted in this system clusters nodes into groups after the relational graph neural network, and performs normalization operation on each group of nodes separately to increase the similarity between feature nodes in the same group, and separates the node representations of different groups. Based on the superior performance of the relational graph convolutional neural network multimodal dialogue emotion recognition model on the IEMOCAP dataset four-class classification, it alleviates the problem of confusion in the recognition of similar emotion categories to a certain extent.

[0033] In this embodiment, the emotion determination module further includes an update unit. The update unit extracts graph structure features of the relationship graph through the feature extraction layer of the DRG network, and updates the nodes according to the relationship type of the directed edges and the information of adjacent utterance nodes. The update formula for the update unit is as follows: ;in, For the first Nodes after convolution The characteristic representation of R is a set of relation types. For nodes The set of neighboring nodes under relation r The normalization coefficient is... and These are the learnable convolution weight matrices. Used for processing neighbor information. Used to process its own residual information. This is the activation function.

[0034] By updating the formula, each node absorbs the sentiment / feature information of its neighbors, thus initially capturing the relationships in the graph structure.

[0035] Facial expressions are a dynamic process, but they also possess some static features. Therefore, to improve recognition accuracy, in one embodiment, the feature extraction unit further includes extracting facial expression features from video data containing facial expressions using bi-branch features. The extraction process includes: Step A: Slice and sample the video data to obtain multiple video segments, including sampling the video at fixed time intervals (e.g., 1 frame per second), dividing the long video into multiple "slices" (i.e., discrete video frame sequences), each slice containing several consecutive frames (e.g., slice i contains several frames before and after the i-th second), transforming the dynamic video into a static image sequence that can be processed frame by frame, thereby reducing computational complexity. Step B: Perform dual-branch feature extraction, including: Step B1: Extract optical flow images from the video segments. The size of the optical flow images is: Optical flow images are used to reflect inter-frame motion information in video frames, such as the trajectory of facial muscle movements. Pre-trained temporal CNN networks are used to extract dynamic facial expression features from optical flow images, such as eyebrow movement when frowning and dynamic changes in the corners of the mouth when smiling. The key architecture of temporal CNNs includes: The first part (from Conv1a to Pool1): large convolutional kernels The dimensionality is quickly compressed, and the pooling layer enhances the robustness of motion features. In the latter part (Conv5a to fc7), small convolutional kernels (3×3) are used to extract local facial movements (such as eyelid blinking and mouth twitching), and finally output a 4096-dimensional vector to encode the dynamic expression information of the optical flow map. Step B2: Crop a photo of the speaker's face from the video segment. The size of the cropped photo is [size to be filled in]. The method focuses on the speaker's static facial structure using a facial photograph and extracts static facial features using a pre-trained spatial CNN network. The spatial CNN and the optical flow branch share the CNN structure to ensure that the dimensions and semantic spaces of static and dynamic features are aligned, which facilitates subsequent fusion and outputs a 4096-dimensional vector. Step B3: Use a pre-trained Deep Belief Network (DBN) to fuse dynamic facial expression features and static facial features, and output facial expression features, including: concatenating the 4096-dimensional vector output by the two branches into an 8192-dimensional vector, and inputting it into the DBN fusion network; Optical flow images can reflect motion information between video frames. Temporal CNN networks, based on dynamic facial expression features extracted from optical flow images, can capture the movement trajectory of facial muscles, such as the changes in the corners of the mouth and the furrowing of eyebrows. Meanwhile, spatial CNN networks extract static facial features from cropped facial photographs, obtaining static information such as the shape of eyebrows and eyes. Combining the two can comprehensively cover the information contained in facial expressions. Compared with a single feature extraction method, it will not miss important expression cues, thus providing a more complete description of facial expressions.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multimodal dialogue emotion recognition system based on relational graph convolutional neural networks, characterized in that: This includes a data acquisition module, a feature determination module, and an emotion determination module that are connected in sequence via communication. The data acquisition module is used to acquire video data of the dialogue to be identified, wherein the video data includes audio data of the dialogue between several parties: The feature determination module determines the emotional features corresponding to each dialogue statement in the dialogue audio data based on the dialogue audio data, and determines the emotional correlation between each dialogue statement based on the context between each dialogue statement. The emotion determination module determines the emotion type corresponding to each statement in the dialogue to be identified based on the emotional correlation between each dialogue statement.

2. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 1, characterized in that: The feature determination module includes a feature extraction unit, a first feature fusion unit, and a second feature fusion unit: The feature extraction unit is used to extract the audio information, image information and text information carried by the video data, determine the text information corresponding to the video data, and use deep learning feature extraction algorithms to extract the audio features, image features and text features from the feature information of the audio information, image information and text information respectively; The first feature fusion unit performs feature fusion on the audio features, image features, and text features using a feature fusion algorithm to obtain a first fused feature; The second feature fusion unit inputs the first fused feature into multiple parallel classifiers, outputs multiple decision results, and then fuses the multiple decision results again to obtain the second fused feature; The emotional correlation between each dialogue statement is determined based on the second fusion feature.

3. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 2, characterized in that: The method by which the second feature fusion unit obtains the second fused feature includes at least one of averaging, voting, weighting, adaptive enhancement, and dynamic Bayesian network.

4. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 1, characterized in that: The emotion determination module includes using a two-layer DRG network to determine the emotion type corresponding to each statement in the dialogue to be identified based on the emotion correlation between each dialogue statement. The relationship graph includes: using the emotion features corresponding to each dialogue statement as graph nodes, and the dialogue relationship type between speakers as directed edges, and connecting each graph node with directed edges. The graph nodes are input into the DRG network, and after convolution operations of multiple layers of convolutional neural networks, the node representations are obtained. Then, a classifier is used to classify the graph nodes and output the emotion type of each dialogue statement.

5. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 3, characterized in that: The emotion determination module also includes an update unit, which extracts the graph structure features of the relationship graph through the feature extraction layer of the DRG network, and updates the nodes according to the relationship type of the directed edges and the information of adjacent discourse nodes.

6. The multimodal dialogue emotion recognition system based on relational graph convolutional neural network according to claim 5, characterized in that: The update formula for the updated unit is: ； in, For the first Nodes after convolution The characteristic representation of R is a set of relation types. For nodes The set of neighboring nodes under relation r The normalization coefficient is... and These are the learnable convolution weight matrices. Used for processing neighbor information. Used to process its own residual information. This is the activation function.

7. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 4, characterized in that: The DRG network includes a normalization operation on the emotional features of each dialogue statement using standard deviation, mean, training scale, and shift vector parameters.

8. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 5, characterized in that: The DRG network uses GraphTransformer to replace the original attention layer, in order to capture and utilize the structural information of the relationship graph and the information between nodes, as well as the structural information and the relationships between nodes.

9. The multimodal dialogue emotion recognition system based on a relational graph convolutional neural network according to claim 1, characterized in that: The feature extraction unit further includes extracting facial expression features from video data containing facial expressions using bi-branch features. The extraction process includes: The video data is sliced ​​and frame-sampling to obtain multiple video segments; The process involves performing bi-branch feature extraction on multiple video segments, including: extracting optical flow images from the video segments and using a temporal CNN network to extract dynamic facial expression features from the optical flow images. The speaker's face is cropped from video segments, and a spatial CNN network is used to extract static facial features. The Deep Belief Network (DBN) is used to fuse dynamic facial expression features and static facial features to output facial expression features.

Images