A multimodal dialogue emotion recognition method and system based on dynamic context-dependent embedding graph neural networks.

Abstract

This invention discloses a multimodal dialogue emotion recognition method and system based on dynamic context dependency embedded graph neural network. First, feature extraction module obtains feature information of different modalities, and pre-trained model contains more useful information. Then, single-modal context dependency is dynamically captured by single-modal graph neural network. Multimodal fusion features are constructed by feature concatenation. Multimodal graph neural network is used to dynamically capture context dependency, providing two graph neural networks to choose from. Finally, emotion recognition category is obtained by emotion classifier. By aggregating context features, the generalization is better and the recognition rate is higher.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a multimodal dialogue emotion recognition method and system based on dynamic context-dependent embedded graph neural networks. Background Technology

[0002] Emotion Recognition in Conversation (ERC) is a key technology in the field of human-computer interaction, with significant value for applications such as mental health monitoring, intelligent assistants, and customer service. As a core element of human communication, emotion influences an individual's attention, memory, and decision-making abilities; accurately identifying emotional changes in conversations can significantly improve the human-computer interaction experience.

[0003] Multimodal emotion recognition, by fusing text, speech, and visual information, can comprehensively capture subtle changes in emotional expression. "Dialogue," as a fundamental form of daily communication, encompasses three modalities: image, text, and audio, and exhibits continuous emotional changes, making it an ideal medium for emotion recognition research. Emotions in dialogue depend not only on the content of the current statement but also on the overall context of the dialogue. Correctly associating preceding and following statements with underlying emotions is crucial for timely detection of emotional fluctuations and shifts, and for achieving accurate recognition. Existing dialogue emotion recognition technologies are primarily based on Long Short-Term Memory (LSTM) networks and Transformer architectures. While LSTM can handle sequential data, it is prone to the vanishing gradient problem in long dialogue contexts, making it difficult to retain early information. Transformer improves long-distance dependency issues through self-attention mechanisms, but its expressive power remains limited for structured data like dialogue. However, existing methods face two key challenges: first, the fixed-window processing mechanism leads to the loss of contextual information, disrupting the coherence of the dialogue; second, speaker transitions and temporal changes in dialogue cannot be effectively modeled, ignoring the intrinsic connection between emotional changes and the dialogue structure.

[0004] Currently, there is a lack of methods to effectively integrate multimodal information and dynamically capture dialogue contextual dependencies. In particular, how to model contextual features separately for image / text / audio modalities, and then further capture cross-modal interaction relationships while ensuring no loss of global information remains a critical problem to be solved. Graph Neural Networks (GNNs) have a natural advantage in representing non-Euclidean data structures, making them particularly suitable for modeling complex dependencies in dialogue scenarios. Therefore, developing a multimodal dialogue emotion recognition method based on graph neural networks that can dynamically capture contextual dependencies has significant theoretical and practical implications for improving the emotional understanding capabilities of human-computer interaction. Summary of the Invention

[0005] Purpose of the invention: This invention provides a multimodal dialogue emotion recognition method and system based on dynamic context-dependent embedding graph neural networks, which aggregates contextual features, resulting in better generalization and higher recognition rate.

[0006] Technical solution: The present invention provides a multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks, comprising the following steps:

[0007] Step 1: Obtain video samples and transcribed text samples from the dataset, and extract text, speech, and visual modal information as t, a, and v, respectively;

[0008] Step 2: Using pre-trained BERT, wav2vec2, and MTCNN models, extract features from text, speech, and visual samples respectively to obtain sentiment features t', a', and v'.

[0009] Step 3: Input the three modal sentiment features into the graph neural network, perform single-modal training on their context dependencies, and further extract the text modal features t”, speech modal features a”, and visual modal information v” that are fused from the context dynamic distance;

[0010] Step 4: After concatenating the multimodal features, obtain multimodal feature x, and re-input it into the multimodal graph neural network for training to obtain the context-dependent sentiment feature x' of multimodal fusion;

[0011] Step 5: Jointly train the three single-modal graph neural networks and the multimodal graph neural network to obtain the optimal model parameters. The graph neural network can be implemented using either a graph convolutional network (GCN) or a graph attention network (GAT).

[0012] Step 6: Input the emotional video sample to be identified into the pre-trained feature extraction network, and then input it into the trained single-modal graph neural network and multi-modal graph neural network to identify the emotion category of the speech sample.

[0013] Furthermore, in step 1, obtaining video samples and transcribed text samples from the dataset, and extracting text, speech, and visual modal information t, a, and v respectively, specifically includes the following steps:

[0014] Step 11: Obtain the transcribed text sample and assemble it into a transcribed file according to the format "speaker ID-text-gender mapping-emotion tag". The speaker ID contains "dialogue ID-speaker ID" and the gender mapping can be a specific number / letter to obtain the corresponding text sample t.

[0015] Step 12: Extract the corresponding speech sample a from the dataset based on the transcribed text sample;

[0016] Step 13: For the corresponding transcribed text samples, extract the corresponding video image samples from the dataset, and extract video frames as visual samples v at specific intervals.

[0017] Furthermore, in step 2, pre-trained BERT, wav2vec2, and MTCNN models are used to extract features from text, speech, and visual samples, respectively, to obtain sentiment features t', a', and v'. Specifically, this includes the following steps:

[0018] Step 21: For the extracted text samples, use the pre-trained BERT model to extract text features t′, and divide the training set and test set according to the dialogue level based on a fixed partitioning protocol.

[0019] Step 22: For the extracted speech samples, use the pre-trained Wav2Vec2 model to extract speech features a' and align the text to divide the dataset.

[0020] Step 23: For the extracted visual samples, use the pre-trained MTCNN model and the MTCNN face detection algorithm to extract the bounding boxes of the faces, extract facial action units through the facial action coding system, extract visual features v', and align the text to divide the dataset.

[0021] Furthermore, in step 3, the three modalities of emotion features are input into the graph neural network, and their contextual dependencies are trained using a single modality. This further extracts text modality features t”, speech modality features a”, and visual modality information v”, which are fused from the contextual dynamic distance. Specifically, this includes the following steps:

[0022] Step 31: Based on the number of dialogues M, construct a dialogue statement mask and align the structural information of different batches of the diagram below to facilitate subsequent unified processing;

[0023] Step 32: Encode the feature sequence using a graph neural network (GNN) and capture the contextual dependencies in the dialogue. Each statement in each dialogue is treated as a node, generating the initial node representation u of the statement.

[0024] Step 33: Using node representation u, construct a graph neural network containing dialogue context information, presented as adjacency matrix adj and edge index information, including node information, node connection relationships and adjacency edge types. The window size is variable and can capture key dialogue information.

[0025] Step 34: Output the single-modal sentiment features t”, a”, c”.

[0026] Furthermore, in step 4, the multimodal features are concatenated to obtain multimodal feature x, which is then re-input into the multimodal graph neural network for training to obtain the context-dependent sentiment feature x' of multimodal fusion. This specifically includes the following steps:

[0027] Step 41: Set the single-modal graphical neural network to inference mode, perform inference on the original samples, and obtain single-modal features;

[0028] Step 42: Concatenate the single-modal features in the order of "text-speech-video" to obtain the multimodal feature x;

[0029] Step 43: Encode the feature sequence through a graph neural network (GNN), repeat the single-modal network operation, capture the contextual dependencies in the dialogue, treat each sentence in each dialogue as a node, and generate the initial node representation u' of the sentence.

[0030] Step 44: Using node representation u', construct a graph neural network containing dialogue context information, presented as adjacency matrix adj' and edge index information edge index', including node information, node connection relationships and the type of adjacent edges. The window size is also variable, which can capture key dialogue information.

[0031] Step 45: Output the multimodal sentiment feature x' of the node and output the prediction scores for each category; In the sentiment classification task, use the classification module to perform multi-category sentiment classification prediction on the node feature x' and output the final sentiment classification result;

[0032] Step 46: The classification module directly selects the category with the highest node prediction score as the emotion classification result.

[0033] Furthermore, in step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network selected is the Graph Convolutional Network (GCN). The specific operation of a single GCN layer is as follows: inputting unimodal samples, constructing a sample matrix input, and performing a linear transformation according to the edge weights, which are learned iteratively; multiplying the adjacency matrix adj with the linearly transformed feature matrix to aggregate the feature information of each node and its neighboring nodes, realizing the propagation and sharing of information in the graph structure; adding the bias vector bias to the output to perform a translation operation on the output features, increasing the expressive power of the model.

[0034] Furthermore, the graph neural network uses a graph convolutional network (GCN). Specifically, the GCN is as follows: the input feature matrix is ​​subjected to graph convolution to obtain first-level features; the first-level features are processed by an activation function and output, and then summed with the original output through a residual connection as the input for the next level of graph convolution; this "GCN-ReLU-residual connection" is repeatedly used to build a five-layer network architecture; the output nodes have single-modal / multi-modal sentiment features.

[0035] Furthermore, in step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network selected is the Graph Attention Network (GAT). The specific operation of a single GAT layer is as follows: inputting a unimodal sample, constructing the node feature matrix x and edge index matrix edge index of the sample, with the edge connection relationship defined by the graph structure; each edge calculates its attention coefficient relative to its neighboring nodes through an attention mechanism, and performs a weighted summation of the features of the neighboring nodes to achieve adaptive information propagation; the attention weights are learned through a parameterized feedforward network, which can support multi-head attention mechanisms. Optionally, the outputs of multiple heads can be concatenated or averaged; the Dropout operation can be applied to the input and attention weights to improve the model's generalization ability.

[0036] Furthermore, the graph neural network selected is the Graph Attention Network (GAT). The GAT model is constructed as follows: the input feature matrix is ​​subjected to the first layer of multi-head graph attention convolution to obtain the first-level node features; the first-level features are processed by the activation function ELU and then input into the second layer of graph attention convolution; the output of the second layer is activated by ELU and then dropped out, and then fed into the output graph attention layer to output the final classification features; the last layer adopts single-head attention output with the same dimension as the number of categories; the output node's single-modal / multi-modal sentiment features are displayed.

[0037] Accordingly, a multimodal dialogue emotion recognition system based on dynamic context-dependent embedding graph neural networks includes: a feature extraction module for extracting single-modal sample features from text, speech, and vision;

[0038] The single-modal graphical neural network module is used to train single-modal sample features and capture the dynamic distance dependencies of sentence context;

[0039] The feature concatenation module is used to concatenate single-modal training features to achieve multi-modal feature fusion.

[0040] The multimodal graph neural network module is used to train multimodal fusion features and capture the dynamic distance dependencies of multimodal feature context.

[0041] The sentiment classification module is used to perform sentiment discrimination and classification on the fused multimodal features.

[0042] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The present invention first obtains feature information of different modalities through a feature extraction module, and the pre-trained model contains more useful information. Then, it dynamically captures the unimodal contextual dependencies through a single-modal graph neural network, constructs multimodal fusion features through feature concatenation, dynamically captures contextual dependencies through a multimodal graph neural network, and provides two graph neural networks to choose from. Finally, it obtains the emotion recognition category through an emotion classifier, aggregates contextual features, and has better generalization and higher recognition rate. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0044] Figure 2 This is an optional graph neural network architecture diagram in this invention.

[0045] Figure 3 This invention utilizes the confusion matrix of a graph convolutional network on the IMEOCAP dataset.

[0046] Figure 4 This invention utilizes the confusion matrix of a graph attention network on the IMEOCAP dataset.

[0047] Figure 5 This invention utilizes the confusion matrix of graph convolutional networks on the MOSI dataset.

[0048] Figure 6 This invention utilizes the confusion matrix of graph attention networks on the MOSI dataset. Detailed Implementation

[0049] like Figure 1 As shown, a multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks includes the following steps:

[0050] (1) Obtain video samples and transcribed text samples from the dataset, and extract text, speech, and visual modal information as t, a, and v, respectively. This step specifically includes:

[0051] (1-1) Obtain the transcribed text sample and assemble it into a transcribed file according to the format "speaker ID-text-gender mapping-emotion tag", where the speaker ID contains "dialogue ID-speaker ID" and the gender mapping is a specific number / letter, and obtain the corresponding text sample t; (1-2) For the corresponding transcribed text sample, extract the corresponding speech sample a from the dataset and align the data volume with the text sample; (1-3) For the corresponding transcribed text sample, extract the corresponding video image sample from the dataset, extract video frames as visual samples v at a specific interval and align the data volume with the text sample.

[0052] In this embodiment, the speaker ID style example for the IMEOCAP dataset is “Ses01M_impro05_F000”, where “Ses01M_impro05” is a unique ID for each dialogue turn, “F” represents “Female”, and “000” represents the speaker sequence number. A similar format is “Ses02M_script01_3_M033”. The IMEOCAP dataset contains 7433 transcribed texts, with males mapped to the number 93 and females to the number 91. In this example, the speaker ID style example for the MOSI dataset is “03bSnISJMiM_13”, where “03bSnISJMiM” represents a unique ID for each dialogue turn, and the number “13” represents the speaking order in the dialogue. The MOSI dataset contains 2199 transcribed texts.

[0053] (2) Using pre-trained BERT, wav2vec2, and MTCNN models, features are extracted from text, speech, and visual samples respectively to obtain sentiment features t', a', and v'. This step specifically includes:

[0054] (2-1) For the extracted text samples, the pre-trained BERT model is used to extract text features. Based on the fixed-split protocol, the training set and test set are divided at the dialogue level. In this example, the IMEOCAP dataset is divided into training and test sets at the dialogue level in an 8:2 ratio. The training set includes 120 dialogues and 5810 sentences, while the test set includes 34 dialogues and 1623 sentences. In this example, the MOSI dataset is divided into training and test sets at the dialogue level in a 2:1 ratio. The training set includes 62 dialogues and 1447 sentences, while the test set includes 31 dialogues and 752 sentences.

[0055] (2-2) For the extracted speech samples, the audio sampling rate was set to 16000Hz. The pre-trained Wav2Vec2 model was used to extract speech features, and the dataset was aligned with the text partitioning.

[0056] (2-3) For the extracted visual samples, the bounding boxes of the faces are extracted using the MTCNN face detection algorithm, facial action units are extracted using the facial action coding system, visual features are extracted, and the dataset is divided into text segments.

[0057] In this embodiment, for the IMEOCAP dataset, the BERT model uses a pre-trained bert-base-uncased model, which outputs a 768-dimensional feature vector; the Wav2Vec2 model uses a pre-trained wav2vec2-base model, which outputs a 768-dimensional feature vector; the MTCNN face detection algorithm extracts 17 facial feature points, which, combined with facial action units, output a 512-dimensional visual feature vector.

[0058] In this embodiment, for the MOSI dataset, an open-source database is used. Text features are extracted and output as 100-dimensional feature vectors, speech features are extracted and output as 73-dimensional feature vectors, and visual features are extracted and output as 100-dimensional feature vectors.

[0059] (3) Input the three modal sentiment features into the graph neural network, perform single-modal training on their context dependencies, and further extract the text modal features t”, speech modal features a”, and visual modal information v” that are fused with the context dynamic distance; this step specifically includes: (3-1) Construct a dialogue statement mask according to the number of dialogues M, align the structural information of different batches of the graph, so as to facilitate subsequent unified processing; (3-2) Encode the feature sequence through the graph neural network (GNN) and capture the context dependencies in the dialogue. Each statement in each dialogue is used as a node to generate the initial node representation of the statement; (3-3) Use the node representation to construct a graph neural network containing dialogue context information, which is presented as adjacency matrix and edge index information, including node information, node connection relationship and type of adjacent edge, etc. In particular, the window size is variable, which can capture key dialogue information; (3-4) Output single-modal sentiment features t”, a”, v”.

[0060] In this embodiment, to capture dynamic dependencies in the dialogue context, a fully connected adjacency matrix with Gaussian decay is used to construct the graph, where the Gaussian decay parameter is sigma, that is, for any two nodes i and j, their edge weight is calculated using the following formula:

[0061]

[0062] Where dist = abs(ij) represents the distance between nodes. This construction method is superior to a fixed window, as it can dynamically adjust the degree of association between nodes based on their distance.

[0063] In this embodiment, sigma is set to 1.5, which can be adjusted through parameter tuning.

[0064] (4) After concatenating the multimodal features, we obtain multimodal feature x, which is then re-input into the multimodal graph neural network for training to obtain the context-dependent sentiment feature x' of multimodal fusion.

[0065] The specific steps include: (4-1) Setting the unimodal graph neural network to inference mode and inferring from the original samples to obtain unimodal features; (4-2) Concatenating the unimodal features in the order of "text-speech-video" to obtain multimodal features x; (4-3) Encoding the feature sequence through a graph neural network (GNN), repeating the unimodal network operation to capture the contextual dependencies in the dialogue, with each sentence in each dialogue as a node, generating the initial node representation x'; (4-4) Using the node representation x', constructing a graph neural network containing dialogue context information, presented as an adjacency matrix and edge index information edge index', including node information, node connection relationships, and the type of adjacent edges, etc. In particular, the window size is also variable, which can capture key dialogue information; (4-5) Outputting the node multimodal sentiment features x' and outputting the prediction scores for each category. In the emotion classification task, the classification module is used to perform multi-category emotion classification prediction on the node features and output the final emotion classification result; (4-6) The classification module directly selects the category with the largest node prediction score as the emotion classification result. Since these are not dialogue output tags, they do not require pooling or fully connected layer operations.

[0066] In this embodiment, the dimension of the concatenated multimodal features is the sum of the dimensions of the three unimodal features. The architecture of the multimodal graph neural network is consistent with that of the unimodal graph neural network to ensure consistency in processing. This network architecture module is adjustable.

[0067] (5) Jointly train the three unimodal graph neural networks and the multimodal graph neural network to obtain the optimal model parameters. For example... Figure 2 As shown, the implementation methods of graph neural networks include optional graph convolutional networks (GCN) and graph attention networks (GAT).

[0068] In this embodiment, if a Graph Convolutional Network (GCN) is used as the implementation of the graph neural network, the optimizer is Adam, the learning rate is set to 0.0005, the number of training epochs is 200, and the loss function is the cross-entropy loss function. If a four-class classification is performed on the IMEOCAP dataset, the training input dimension is 110 dimensions (i.e., the number of graph nodes), the hidden layer dimension is 64 dimensions, and the output layer dimension is 4 dimensions (i.e., the number of output classes). The output confusion matrix is ​​as follows. Figure 3 As shown; if binary classification is performed on the MOSI dataset, the training input dimension is 110 dimensions, i.e., the number of graph nodes, the hidden layer dimension is 64 dimensions, and the output layer dimension is 4 dimensions, i.e., the number of output classes. The output confusion matrix is ​​as follows. Figure 5 As shown.

[0069] In this embodiment, if a Graph Attention Network (GAT) is used as the implementation of the graph neural network, the optimizer is Adam, the learning rate is set to 0.0005, the training epochs are 200, and the loss function is the cross-entropy loss function. If a four-class classification is performed on the IMEOCAP dataset, the first layer training input dimension is 110 dimensions (i.e., the number of graph nodes), the hidden layer dimension is 64 dimensions, and a multi-head attention mechanism is introduced with 4 heads; the second layer training input dimension is the number of graph nodes × the number of attention projections, i.e., 110 * 4 = 440 dimensions, the hidden layer dimension is 64 dimensions, and the number of multi-head attention projections is 4; the third layer, the graph attention layer, is the output layer, with a training input dimension of the number of graph nodes × the number of attention projections, i.e., 110 * 4 = 440 dimensions, a single-head attention mechanism, and an output dimension of 4, i.e., the number of emotion categories. Throughout the implementation, dropout = 0.5 is used. The output confusion matrix is ​​as follows. Figure 4 As shown.

[0070] If binary classification is applied to the MOSI dataset, the first training layer has a 110-dimensional input (graph node count) and a 64-dimensional hidden layer, employing a multi-head attention mechanism with 4 attention heads. The second training layer has a 440-dimensional input (graph node count × attention head count), with a 64-dimensional hidden layer and 4 multi-head attention heads. The third graph attention layer is the output layer, with a 440-dimensional input (graph node count × attention head count), employing a single-head attention mechanism, and an output dimension of 2 (number of emotion categories). Throughout the implementation, dropout = 0.5 is used.

[0071] If a Graph Attention Network (GAT) is used, the attention-encoded features need to be input into a fully connected linear layer for linear mapping transformation. For effective utterances, the category score with the highest attention score is obtained, and the category is output, resulting in a confusion matrix as shown below. Figure 6 As shown.

[0072] Finally, the classification accuracy and F1 score of the experiment in different datasets are shown in Table 1.

[0073] Table 1 Comparison of experimental results on IMEOCAP and MOSI datasets.

[0074]

[0075] (6) Input the emotional video sample to be identified into the pre-trained feature extraction network for reasoning, and then input it into the trained single-modal graph neural network and multi-modal graph neural network to identify the emotion category of the voice sample.

[0076] In this embodiment, during testing, the test samples are directly passed sequentially through a pre-trained feature extraction network, a unimodal graph neural network, and a multimodal graph neural network for forward inference to obtain unimodal emotion classification features; after concatenation, multimodal fusion features are obtained and input into the multimodal network for training.

[0077] This embodiment also provides a multimodal dialogue emotion recognition system based on dynamic context-dependent embedding graph neural networks, including:

[0078] The feature extraction module is used to extract features from single-modal samples of text, speech, and vision.

[0079] The single-modal graph neural network module is trained on single-modal sample features to capture dynamic distance dependencies in the context of sentences.

[0080] The feature concatenation module concatenates single-modal training features for multimodal feature fusion.

[0081] The multimodal graph neural network module is trained on multimodal fusion features to capture the dynamic distance dependencies of multimodal feature context.

[0082] The sentiment classification module is used to classify and discriminate sentiment based on multimodal features.

[0083] The feature extraction module specifically includes: a text feature extraction unit, which processes text using a pre-trained BERT model; a speech feature extraction unit, which samples the speech signal and processes the speech using a pre-trained Wav2Vec2 model; and a visual feature extraction unit, which extracts frames from the complete video, extracts the bounding boxes of the face using the MTCNN face detection algorithm, and extracts facial action units through a facial action coding system.

[0084] The single-modal graph neural network module includes three single-modal context graph neural networks (GNNs) for text, speech, and vision. The specific implementation methods can be graph convolutional network (GCN) or graph attention network (GAT): graph convolutional network units are stacked with five GCN layers to introduce residual connections; graph attention network is stacked with two GAT layers and adds multi-dimensional attention to aggregate neighbor node information.

[0085] The feature concatenation module specifically includes: a single-modal inference unit, which re-inputs the sentiment sample to obtain single-modal network features; and feature concatenation, which forms new features to complete multimodal fusion.

[0086] The multimodal graph neural network module is implemented in the same way as the single-modal graph neural network module, specifically including optional graph convolutional networks (GCN) and graph attention modules (GAT).

[0087] The sentiment classification module includes: a node prediction unit, which outputs a predicted sentiment score for each node; and a sentiment classification unit, which converts the predicted scores into classification categories.

Claims

1. A multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks, characterized in that, Includes the following steps: Step 1: Obtain video samples and transcribed text samples from the dataset, and extract text, speech, and visual modal information as t, a, and v, respectively; Step 2: Using pre-trained BERT, wav2vec2, and MTCNN models, extract features from text, speech, and visual samples respectively to obtain sentiment features. , , ; Step 3: Input the three modalities of sentiment features into the graph neural network, perform single-modal training on their context dependencies, and further extract text modal features that incorporate context dynamic distance. Speech modal features Visual modal information ; Step 4: After concatenating the multimodal features, obtain multimodal feature x, and re-input it into the multimodal graph neural network for training to obtain the context-dependent sentiment feature x' of multimodal fusion; Specifically, the steps include the following: Step 41: Set the single-modal graphical neural network to inference mode, perform inference on the original samples, and obtain single-modal features; Step 42: Concatenate the single-modal features in the order of "text-speech-video" to obtain the multimodal feature x; Step 43: Encode the feature sequence using a graph neural network (GNN), repeat the unimodal network operation, capture the contextual dependencies in the dialogue, and generate the initial node representation of each statement in each dialogue segment as a node. '; Step 44: Represent using nodes 'Construct a graph neural network that includes dialogue context information, using an adjacency matrix.' The edge index information is presented, including node information, node connection relationships and the type of adjacent edges. The window size is also variable, which can capture key dialogue information. Step 45: Output the multimodal sentiment features x' of the node and output the prediction scores for each category; in the sentiment classification task, the classification module is used to analyze the node features. Perform multi-category emotion classification prediction and output the final emotion classification result; Step 46: The classification module directly selects the category with the highest node prediction score as the sentiment classification result; Step 5: Jointly train the three single-modal graph neural networks and the multimodal graph neural network to obtain the optimal model parameters. The graph neural network can be implemented using either a graph convolutional network (GCN) or a graph attention network (GAT). Step 6: Input the emotional video sample to be identified into the pre-trained feature extraction network, and then input it into the trained single-modal graph neural network and multi-modal graph neural network to identify the emotion category of the speech sample.

2. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, Step 1 involves obtaining video samples and transcribed text samples from the dataset, and extracting text, speech, and visual modal information as t, a, and v, respectively. The specific steps include: Step 11: Obtain the transcribed text sample and assemble it into a transcribed file according to the format "speaker ID-text-gender mapping-emotion tag". The speaker ID contains "dialogue ID-speaker ID" and the gender mapping can be a specific number / letter to obtain the corresponding text sample t. Step 12: Extract the corresponding speech sample a from the dataset based on the transcribed text sample; Step 13: For the corresponding transcribed text samples, extract the corresponding video image samples from the dataset, and extract video frames as visual samples v at specific intervals.

3. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, In step 2, pre-trained BERT, wav2vec2, and MTCNN models are used to extract features from text, speech, and visual samples, respectively, to obtain sentiment features. , , Specifically, it includes the following steps: Step 21: For the extracted text samples, use a pre-trained BERT model to extract text features. Based on a fixed partitioning protocol, the training set and test set are divided according to the dialogue level; Step 22: For the extracted speech samples, use the pre-trained Wav2Vec2 model to extract speech features. Align the text to divide the dataset; Step 23: For the extracted visual samples, use the pre-trained MTCNN model and the MTCNN face detection algorithm to extract the bounding boxes of the faces, extract facial action units through the facial action coding system, and extract visual features. Align the text to divide the dataset.

4. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, In step 3, the three modalities of sentiment features are input into the graph neural network, and their context dependencies are trained using a single modality. This further extracts text modality features that incorporate context dynamic distance. Speech modal features Visual modal information Specifically, it includes the following steps: Step 31: Based on the number of dialogues M, construct a dialogue statement mask and align the structural information of different batches of the diagram below to facilitate subsequent unified processing; Step 32: Encode the feature sequence using a Graph Neural Network (GNN) and capture the contextual dependencies in the dialogue. Each statement in each dialogue segment is treated as a node to generate the initial node representation of the statement. ; Step 33: Represent using nodes Construct a graph neural network that includes dialogue context information, using an adjacency matrix. The edge index information is presented, including node information, node connection relationships, and the type of adjacent edges. The window size is variable and can capture key dialogue information. Step 34: Output single-modal sentiment features , , .

5. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, In step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network selected is the Graph Convolutional Network (GCN). The specific operation of a single GCN layer is as follows: inputting unimodal samples, constructing a sample matrix input, and performing a linear transformation according to the edge weights, which are learned iteratively; multiplying the adjacency matrix adj with the linearly transformed feature matrix to aggregate the feature information of each node and its neighboring nodes, realizing the propagation and sharing of information in the graph structure; adding the bias vector bias to the output to perform a translation operation on the output features, increasing the expressive power of the model.

6. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 5, characterized in that, The graph neural network uses a graph convolutional network (GCN). Specifically, the GCN is as follows: input feature matrix, perform graph convolution operation to obtain first-level features; output the first-level features through an activation function, and sum them with the original output through a residual connection as the input for the next level graph convolution operation; repeat the five-layer network architecture; output nodes with single-modal / multi-modal sentiment features.

7. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, In step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network selected is the Graph Attention Network (GAT). The specific operation of a single GAT layer is as follows: inputting unimodal samples, constructing the node feature matrix x and edge index matrix of the samples, and the connection relationship of the edges is defined by the graph structure; each edge calculates its attention coefficient relative to its neighboring nodes through the attention mechanism, and performs a weighted summation of the features of the neighboring nodes to achieve adaptive information propagation; the attention weights are learned through a parameterized feedforward network, supporting a multi-head attention mechanism, concatenating or averaging the outputs of multiple heads; the Dropout operation is applied to the input and attention weights to improve the model's generalization ability.

8. The multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 7, characterized in that, The graph neural network selected is the Graph Attention Network (GAT). The GAT model is built as follows: the input feature matrix is ​​subjected to the first layer of multi-head graph attention convolution to obtain the first-level node features; the first-level features are processed by the activation function ELU and then input into the second layer of graph attention convolution; the output of the second layer is activated by ELU and then dropped out and sent into the output graph attention layer to output the final classification features. The final layer uses single-head attention output, with the same number of dimensions and categories; it outputs the unimodal / multimodal sentiment features of the nodes.

9. A system based on the multimodal dialogue emotion recognition method based on dynamic context-dependent embedding graph neural networks as described in claim 1, characterized in that, include: The feature extraction module is used to extract features from single-modal samples of text, speech, and vision. The single-modal graphical neural network module is used to train single-modal sample features and capture the dynamic distance dependencies of sentence context; The feature concatenation module is used to concatenate single-modal training features to achieve multi-modal feature fusion. The multimodal graph neural network module is used to train multimodal fusion features and capture the dynamic distance dependencies of multimodal feature context. The sentiment classification module is used to perform sentiment discrimination and classification on the fused multimodal features.

Images