Multi-modal dialogue emotion recognition method and system based on dynamic context dependency relationship embedded graph neural network

Abstract

The invention discloses a multi-modal dialogue emotion recognition method and system based on a dynamic context dependency embedded graph neural network, and the method comprises the steps: firstly obtaining the feature information of different modals through a feature extraction module, and enabling a pre-training model to comprise more beneficial information; then dynamically capturing a single-mode context dependency relationship through a single-mode graph neural network, constructing a multi-mode fusion feature through feature splicing, dynamically capturing a context dependency relationship through a multi-mode graph neural network, providing two selectable graph neural networks, and finally obtaining an emotion recognition category and aggregating context features through an emotion classifier. The generalization is better, and the recognition rate is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, in particular to a multi-modal dialogue emotion recognition method and system based on dynamic context dependency relationship embedded graph neural network. BACKGROUND

[0002] Emotion Recognition in Conversation (ERC) is a key technology in human-computer interaction, which has important value for mental health monitoring, intelligent assistants, customer service, etc. Emotion, as the core element of human communication, affects an individual's attention, memory, and decision-making ability. Accurate recognition of emotional changes in conversation can significantly improve the human-computer interaction experience.

[0003] Multi-modal emotion recognition can fully capture subtle changes in emotional expression by fusing text, speech, and visual information. Among them, "conversation" as the basic form of daily communication contains three modalities of graph / text / audio, and has the characteristics of continuous emotional changes, making it an ideal carrier for emotion recognition research. The emotion in the conversation not only depends on the content of the current sentence, but also is closely related to the context of the entire conversation. Correctly associating the previous and subsequent sentences with the underlying emotion is conducive to timely discovering the fluctuations and changes in emotions, and is crucial for accurate recognition. Existing dialogue emotion recognition techniques are mainly based on Long Short-Term Memory (LSTM) and Transformer architecture. LSTM can handle sequence data, but it is prone to gradient vanishing problem in long dialogue context, making it difficult to retain early information. Transformer improves the long-distance dependency problem through self-attention mechanism, but for structured data such as conversation, its expressive ability is still limited. However, existing methods face two key challenges: first, the fixed window processing mechanism leads to the loss of context information, breaking the continuity of the conversation; second, the speaker transition and temporal changes in the conversation cannot be effectively modeled, ignoring the intrinsic relationship between emotional changes and conversation structure.

[0004] In the prior art, there is still a lack of a method that can effectively integrate multi-modal information and dynamically capture the context dependency relationship of the conversation. In particular, how to model the context features for the three modalities of graph / text / audio, further capture the cross-modal interaction relationship, and ensure that the global information is not lost, is still a key problem that needs to be solved urgently. Graph Neural Network (GNN) has the natural advantage of expressing non-Euclidean data structure, and is particularly suitable for modeling complex dependency relationships in the dialogue scenario. Therefore, developing a multi-modal dialogue emotion recognition method based on graph neural network that can dynamically capture the context dependency relationship has important theoretical and practical significance for improving the emotional understanding ability of human-computer interaction. SUMMARY

[0005] The application provides a multi-modal dialogue emotion recognition method and system based on a dynamic context dependency relationship embedded graph neural network.

[0006] The application discloses a multi-modal dialogue emotion recognition method based on a dynamic context dependency relationship embedded graph neural network.

[0007] Step 1, obtain a data set video sample and a transcribed text sample, and extract text, speech and visual modal information from the video sample and the text sample, and the text, speech and visual modal information are respectively denoted as t, a and v;

[0008] Step 2, a pre-trained BERT, wav2vec2 and MTCNN model are used to extract features of the text, speech and visual samples, and emotion features t', a' and v' are obtained;

[0009] Step 3, the three modal emotion features are input into a graph neural network, the context dependency relationship is trained in a single modal, and text modal features t'', speech modal features a'' and visual modal information v'' that have further extracted and fused the context dynamic distance are obtained;

[0010] Step 4, the multi-modal features are spliced to obtain multi-modal features x, and the multi-modal features x are re-input into a multi-modal graph neural network for training, and context-dependent emotion features x' of multi-modal fusion are obtained;

[0011] Step 5, the three single-modal graph neural networks and the multi-modal graph neural network are jointly trained to obtain optimal model parameters, wherein the graph neural network comprises an optional graph convolution network GCN and a graph attention network GAT;

[0012] Step 6, a to-be-recognized emotion video sample is input into a pre-trained feature extraction network, and then input into the trained single-modal graph neural network and multi-modal graph neural network, so that the emotion category of the speech sample is recognized.

[0013] Further, in step 1, the data set video sample and the transcribed text sample are obtained, and text, speech and visual modal information are extracted from the video sample and the text sample, and the text, speech and visual modal information are respectively denoted as t, a and v, and the step specifically comprises the following steps.

[0014] Step 11, a transcribed text sample is obtained, and the transcribed text sample is assembled into a transcribed file in a "speaker id-text-gender mapping-emotion label" format, wherein the speaker id comprises a "dialogue id-speaker id", the gender mapping is a specific number / letter, and a corresponding text sample t is obtained.

[0015] Step 12, corresponding to the transcribed text sample, a data set corresponding speech sample a is extracted;

[0016] Step 13, corresponding to the transcribed text sample, extract the corresponding video image sample of the data set, and extract video frames as visual samples v according to a certain interval.

[0017] Further, in step 2, pre-trained BERT, wav2vec2, and MTCNN models are used to extract features from text, speech, and visual samples, respectively, to obtain emotion features t', a', and v'. The specific steps include the following:

[0018] Step 21, for the extracted text sample, use the pre-trained BERT model to extract text features t', and based on a fixed partition protocol, divide the training set and test set according to the dialogue level;

[0019] Step 22, for the extracted speech sample, use the pre-trained Wav2Vec2 model to extract speech features a', and align the text to divide the data set;

[0020] Step 23, for the extracted visual sample, use the pre-trained MTCNN model to extract the boundary box of the face using the MTCNN face detection algorithm, extract the facial action unit through the facial action coding system, extract the visual feature v', and align the text to divide the data set.

[0021] Further, in step 3, the three modal emotion features are input into the graph neural network, and the context dependency relationship is trained in a single modal, and further text modal features t", speech modal features a", and visual modal information v" that have fused the context dynamic distance are extracted. The specific steps include the following:

[0022] Step 31, according to the number of dialogues M, construct a dialogue sentence mask to align the structure information of the graph in different batches, facilitating subsequent unified processing;

[0023] Step 32, encode the feature sequence through the graph neural network GNN and capture the context dependency in the dialogue, with each sentence in each dialogue as a node, to generate the initial node representation u of the sentence;

[0024] Step 33, use the node representation u to construct a graph neural network containing dialogue context information, with the information presented in the adjacency matrix adj and edge index information edge index, including node information, connection relationship of nodes, and type of adjacent edges, with variable window size to capture key dialogue information;

[0025] Step 34, output single-modal emotion features t", a", and c".

[0026] Further, in step 4, after concatenating the multi-modal features, obtain the multi-modal feature x, and re-input it into the multi-modal graph neural network for training to obtain the multi-modal fusion context-dependent emotion feature x'. The specific steps include the following:

[0027] Step 41, set the single-modal graph neural network to inference mode, and perform inference on the original sample to obtain single-modal features;

[0028] Step 42, concatenate the single-modal features in the order of "text-speech-video" to obtain multi-modal features x;

[0029] Step 43, encode the feature sequence through the graph neural network GNN, repeat the single-modal network operation, capture the context dependence in the dialogue, and generate the initial node representation u' of each sentence in each dialogue as a node;

[0030] Step 44, use the node representation u' to build a graph neural network containing dialogue context information, present the information in the form of adjacency matrix adj' and edge index information edge index', including node information, connection relationship of nodes and type of adjacent edges, and the window size is also variable, which can capture key dialogue information;

[0031] Step 45, output the node multi-modal sentiment features x', and output the prediction scores of each category; in the emotion classification task, use the classification module to perform multi-class emotion classification prediction on the node features x', and output the final emotion classification result;

[0032] Step 46, the classification module directly selects the category with the maximum node prediction score as the emotion classification result.

[0033] Further, in step 5, the three single-modal graph neural networks and the multi-modal graph neural network are jointly trained to obtain optimal model parameters. The graph neural network uses a graph convolution network GCN. The specific operation of a single GCN layer is as follows: input the single-modal sample, construct a sample matrix matrix input, perform linear transformation according to the edge weight, and the edge weight is obtained through learning iteration; multiply the adjacency matrix adj by the feature matrix after linear transformation to aggregate the feature information of each node and its neighbor nodes, realize the propagation and sharing of information in the graph structure; add the bias vector bias to the output to perform translation operation on the output features, and increase the expression ability of the model.

[0034] Further, the graph neural network uses a graph convolution network GCN. The graph convolution network GCN is as follows: input the feature matrix, perform graph convolution operation to obtain first-level features; add the first-level features outputted through the activation function to the original output through residual connection to obtain the input of the next-level graph convolution operation; repeat the "GCN-ReLU-residual connection" to build a five-layer network architecture; output the node single-modal / multi-modal sentiment features.

[0035] Furthermore, in step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network uses a graph attention network (GAT). The specific operations of a single GAT layer are as follows: a unimodal sample is input, and the node feature matrix x and edge index matrix edge index of the sample are constructed. The edge connectivity is defined by the graph structure. Each edge calculates the attention coefficient relative to its neighboring nodes through the attention mechanism, and performs a weighted sum of the neighboring node features to achieve adaptive information propagation. The attention weights are learned through a parameterized feedforward network to support multi-head attention mechanisms. Optionally, the outputs of multiple heads are concatenated or averaged. Dropout is applied to the input and attention weights to improve model generalization.

[0036] Furthermore, the graph neural network uses the graph attention network GAT. The specific construction of the attention network GAT model is as follows: input the feature matrix, perform the first-layer multi-head graph attention convolution, and obtain the first-level node features; the first-level features are processed by the activation function ELU, and then input into the second-layer graph attention convolution; the second-layer output is activated and Dropout by ELU, and then sent to the output graph attention layer to output the final classification features; the last layer uses single-head attention output, and the dimension is the same as the number of categories; the unimodal / multimodal sentiment features of the output node.

[0037] Accordingly, a multimodal conversation emotion recognition system based on dynamic context dependency embedding graph neural network includes: a feature extraction module for extracting unimodal sample features of text, speech, and vision;

[0038] A unimodal graph neural network module is used to train unimodal sample features and capture the dynamic distance dependencies of sentence contexts;

[0039] Feature splicing module, used to perform feature splicing on single-modal training features to achieve multi-modal feature fusion;

[0040] Multimodal graph neural network module, used to train multimodal fusion features and capture the dynamic distance dependencies of multimodal feature contexts;

[0041] The sentiment classification module is used to perform sentiment discrimination and classification on the fused multimodal features.

[0042] Beneficial effects: Compared with the prior art, the present application has the following remarkable advantages: firstly, the present application obtains feature information of different modalities through a feature extraction module, and the pre-training model contains more beneficial information; then, the single-modal graph neural network dynamically captures the context dependency relationship on the single modality, the multi-modal fusion features are constructed through feature splicing, the context dependency relationship is dynamically captured through the multi-modal graph neural network, and two optional graph neural networks are provided; finally, the sentiment recognition categories are obtained through the sentiment classifier, and the context features are aggregated, which is better in generalization and higher in recognition rate. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a method flowchart of the present application.

[0044] Figure 2 It is an optional graph neural network architecture diagram in the present application.

[0045] Figure 3 It is a confusion matrix of the present application using graph convolution network under IMEOCAP dataset.

[0046] Figure 4 It is a confusion matrix of the present application using graph attention network under IMEOCAP dataset.

[0047] Figure 5 It is a confusion matrix of the present application using graph convolution network under MOSI dataset.

[0048] Figure 6 It is a confusion matrix of the present application using graph attention network under MOSI dataset. DETAILED DESCRIPTION

[0049] As shown in Figure 1 A multi-modal dialogue emotion recognition method based on dynamic context dependency relationship embedding graph neural network, comprising the following steps:

[0050] (1) obtaining dataset video samples, transcribed text samples, extracting text, speech, visual modal information t, a, v respectively. This step specifically includes:

[0051] (1-1) obtaining transcribed text samples, assembling them into transcribed files according to the "speaker id-text-gender mapping-emotion label" format, wherein the speaker id contains "dialogue id-speaker id", the gender mapping is a specific number / letter, and the corresponding text sample t is obtained; (1-2) corresponding to the transcribed text sample, extracting the corresponding speech sample a of the dataset, aligning the data amount with the text sample; (1-3) corresponding to the transcribed text sample, extracting the corresponding video image sample of the dataset, extracting video frames as visual samples v according to a specific interval, and aligning the data amount with the text sample.

[0052] In this embodiment, the speaker id style example of the IMEOCAP dataset is "Ses01M_impro05_F000", "Ses01M_impro05" is the unique id of the dialogue round, F represents "Female" gender, and "000" represents the speaker number. There are similar formats such as "Ses02M_script01_3_M033". The IMEOCAP dataset has a total of 7433 transcribed texts, with male mapped to the number 93 and female mapped to the number 91. In this example, the speaker id style example of the MOSI dataset is "03bSnISJMiM_13", "03bSnISJMiM" represents the unique id of the dialogue round, and the number "13" represents the speaking order in the dialogue. The MOSI dataset has a total of 2199 transcribed texts.

[0053] (2) Using pre-trained BERT, wav2vec2, MTCNN models, respectively extracting features from text, speech, and visual samples to obtain sentiment features t', a', and v'. This step specifically includes:

[0054] (2-1) For the extracted text samples, use the pre-trained BERT model to extract text features, and divide the training set and test set according to the dialogue level based on the fixed split protocol (fixed-split). In this example, the IMEOCAP dataset is divided into training set and test set at the dialogue level in the ratio of 8:2, the training set includes 120 dialogue rounds and 5810 sentences, and the test set includes 34 dialogue rounds and 1623 sentences. In this example, the MOSI dataset is divided into training set and test set at the dialogue level in the ratio of 2:1, the training set includes 62 dialogue rounds and 1447 sentences, and the test set includes 31 dialogue rounds and 752 sentences.

[0055] (2-2) For the extracted speech samples, the audio sampling rate is set to 16000Hz. Use the pre-trained Wav2Vec2 model to extract speech features, and align the text to divide the dataset;

[0056] (2-3) For the extracted visual samples, use the MTCNN face detection algorithm to extract the bounding box of the face, extract the facial action unit through the facial action coding system, extract the visual features, and align the text to divide the dataset.

[0057] In this embodiment, for the IMEOCAP dataset, the BERT model uses the pre-trained bert-base-uncased model to output a 768-dimensional feature vector; the Wav2Vec2 model uses the pre-trained wav2vec2-base model to output a 768-dimensional feature vector; the MTCNN face detection algorithm extracts 17 facial feature points and combines the facial action unit (Action Units) to output a 512-dimensional visual feature vector.

[0058] In this embodiment, for the MOSI dataset, a database with open source data is used, the text extraction feature outputs a 100-dimensional feature vector, the speech extraction feature outputs a 73-dimensional feature vector, and the visual extraction feature outputs a 100-dimensional feature vector.

[0059] (3) Input the three modal emotion features into the graph neural network, train the context dependency relationship, and further extract the text modal feature t", the speech modal feature a", and the visual modal information v" which are fused with the context dynamic distance; this step specifically includes: (3-1) according to the number of dialogues M, construct a dialogue sentence mask, align the structure information of the graph under different batches, and facilitate subsequent unified processing; (3-2) encode the feature sequence through the graph neural network (GNN) and capture the context dependency in the dialogue, each sentence in each dialogue as a node, generate the initial node representation of the sentence; (3-3) use the node representation to construct a graph neural network containing dialogue context information, present the information in the form of adjacency matrix and edge index information edge index, including node information, connection relationship of nodes and type of adjacent edges, etc. In particular, the window size is variable, which can capture key dialogue information; (3-4) output the single-modal emotion features t", a" and v".

[0060] In this embodiment, in order to capture the dynamic dependency relationship in the dialogue context, a full connection adjacency matrix with Gaussian attenuation is used to construct the graph, wherein the Gaussian attenuation parameter is sigma, that is, for any two nodes i and j, the edge weight is calculated as follows:

[0061]

[0062] Wherein dist = abs (i-j) is the distance between nodes. This construction method is superior to the fixed window, which can dynamically adjust the correlation degree between nodes according to the distance.

[0063] In this embodiment, the value of sigma is 1.5, which can be adjusted by parameter tuning.

[0064] (4) After the multi-modal features are spliced, the multi-modal feature x is obtained, which is re-input into the multi-modal graph neural network for training, and the multi-modal fusion context-dependent emotion feature x' is obtained.

[0065] The step specifically comprises: (4-1) setting the single-modal graph neural network to an inference mode, performing inference on the original sample to obtain single-modal features; (4-2) concatenating the single-modal features in the order of "text-voice-video" to obtain multi-modal features x; (4-3) encoding the feature sequence through a graph neural network (GNN), repeating the single-modal network operation, capturing the context dependence in the dialogue, taking each sentence in each dialogue as a node, and generating sentence initial node representation x'; (4-4) using the node representation x', constructing a graph neural network containing dialogue context information, and presenting information in the form of an adjacency matrix, edge index information edge index', containing node information, connection relationship of nodes and type of adjacent edges, etc. In particular, the window size is also variable, and key dialogue information can be captured; (4-5) outputting node multi-modal sentiment features x', and outputting prediction scores of various categories. In the emotion classification task, a classification module is used to perform multi-class emotion classification prediction on the node features, and finally output the emotion classification result; (4-6) the classification module directly selects the category with the maximum node prediction score as the emotion classification result. Since the dialogue output label is not required, there is no need to go through the pooling and fully connected layer operations.

[0066] In this embodiment, the dimension of the multi-modal feature after splicing is the sum of the dimensions of the three single-modal features. The architecture of the multi-modal graph neural network is consistent with that of the single-modal graph neural network to ensure consistency in processing. The network architecture module can be adjusted.

[0067] (5) Joint training of the three single-modal graph neural networks and the multi-modal graph neural network to obtain optimal model parameters. As shown in Figure 2 The implementation of the graph neural network includes an optional graph convolution network (GCN) and a graph attention network (GAT).

[0068] In this embodiment, if the graph convolution network (GCN) is used as the implementation of the graph neural network, the optimizer is Adam, the learning rate is set to 0.0005, the number of training rounds is 200 rounds, and the loss function is the cross-entropy loss function. If the IMEOCAP dataset is implemented for four classification, the training input dimension is 110, that is, the number of graph nodes, the hidden layer dimension is 64, and the output layer dimension is 4, that is, the number of output categories. The output confusion matrix is as shown in Figure 3 If the MOSI dataset is implemented for two classification, the training input dimension is 110, that is, the number of graph nodes, the hidden layer dimension is 64, and the output layer dimension is 4, that is, the number of output categories. The output confusion matrix is as shown in Figure 5 .

[0069] In this embodiment, if the graph attention network (GAT) is used as the implementation manner of the graph neural network, the optimizer is Adam, the learning rate is set to 0.0005, the number of training rounds is 200 rounds, and the loss function is the cross-entropy loss function. If four classification is implemented on the IMEOCAP dataset, the first layer training input dimension is 110 dimensions, that is, the number of graph nodes, the hidden layer dimension is 64 dimensions, the multi-head attention mechanism is introduced, and the number of heads is 4; the training input dimension of the second layer is the number of graph nodes x the number of attention heads, that is, 110*4=440 dimensions, the hidden layer is 64 dimensions, and the number of multi-head attention heads is 4; the third layer of the graph attention layer is the output layer, the training input dimension is the number of graph nodes x the number of attention heads, that is, 110*4=440 dimensions, the single-head attention mechanism is used, and the output dimension is 4, that is, the number of emotion classification categories. During the implementation process, dropout=0.5 is used. The output confusion matrix is as shown in Table 1. Figure 4

[0070] If the MOSI dataset is implemented for two classification, the first layer training input dimension is 110 dimensions, that is, the number of graph nodes, the hidden layer dimension is 64 dimensions, the multi-head attention mechanism is introduced, and the number of heads is 4; the training input dimension of the second layer is the number of graph nodes x the number of attention heads, that is, 110*4=440 dimensions, the hidden layer is 64 dimensions, and the number of multi-head attention heads is 4; the third layer of the graph attention layer is the output layer, the training input dimension is the number of graph nodes x the number of attention heads, that is, 110*4=440 dimensions, the single-head attention mechanism is used, and the output dimension is 2, that is, the number of emotion classification categories. During the implementation process, dropout=0.5 is used.

[0071] If the graph attention network GAT is used, the features after attention encoding need to be finally input into the linear fully connected layer for linear mapping transformation. For the valid utterance, the class score with the maximum attention score is obtained, the output category is obtained, and the confusion matrix is as shown in Table 1. Figure 6

[0072] Finally, the classification accuracy and F1 score in different datasets are shown in Table 1.

[0073] Table 1 Comparison of experimental results on IMEOCAP and MOSI datasets

[0074]

[0075] (6) The to-be-recognized emotional video sample is input into the pre-trained feature extraction network for inference, and then input into the trained single-modal graph neural network and multi-modal graph neural network to identify the emotion category of the speech sample.

[0076] ​​In this embodiment, during testing, the test sample is directly passed through the pre-trained feature extraction network, the single-modal graph neural network and the multi-modal graph neural network in sequence for forward inference to obtain single-modal emotion classification features; the multi-modal fusion features obtained after splicing are input into the multi-modal network for training.

[0077] The embodiment also provides a multi-modal dialogue emotion recognition system based on a dynamic context dependency relationship embedded graph neural network, which comprises:

[0078] A feature extraction module is configured to extract single-modal sample features of text, speech and vision.

[0079] A single-modal graph neural network module is configured to train the single-modal sample features to capture sentence context dynamic distance dependency relationships.

[0080] A feature splicing module is configured to splice the single-modal training features to perform multi-modal feature fusion.

[0081] A multi-modal graph neural network module is configured to train the multi-modal fusion features to capture multi-modal feature context dynamic distance dependency relationships.

[0082] An emotion classification module is configured to perform emotion classification on the multi-modal features.

[0083] The feature extraction module specifically comprises: a text feature extraction unit configured to process text by using a pre-trained BERT model; a speech feature extraction unit configured to sample a speech signal and process the speech by using a pre-trained Wav2Vec2 model; and a visual feature extraction unit configured to extract frames from a complete video, extract a face bounding box by using an MTCNN face detection algorithm, and extract a facial action unit by using a facial action coding system.

[0084] The single-modal graph neural network module comprises three single-modal context graph neural networks (GNNs) of text, speech and vision, and the optional specific implementation modes are graph convolution networks (GCNs) and graph attention networks (GATs). The graph convolution network unit comprises five stacked GCN layers with residual connections. The graph attention network comprises two stacked GAT layers with multi-element attention aggregation neighbor node information.

[0085] The feature splicing module specifically comprises: a single-modal inference unit configured to re-input an emotion sample to obtain single-modal network features; and a feature splicing unit configured to form new features for completing multi-modal fusion.

[0086] The multi-modal graph neural network module is specifically implemented as the single-modal graph neural network module and specifically comprises optional graph convolution networks (GCNs) and graph attention modules (GATs).

[0087] The sentiment classification module includes a node prediction unit that outputs a predicted sentiment score for each node and a sentiment classification unit that converts the predicted scores into a classification category.

Claims

1. A multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network, characterized by: The steps include: Step 1: Obtain the video samples and transcribed text samples of the dataset, and extract the text, speech, and visual modal information t, a, and v from them respectively; Step 2: Use the pre-trained BERT, wav2vec2, and MTCNN models to extract features from text, speech, and visual samples, respectively, and obtain the sentiment features t', a', and v'; Step 3: Input the three modal sentiment features into the graph neural network, perform single-modal training on their contextual dependencies, and further extract text modal features t”, speech modal features a”, and visual modal information v” that integrate the contextual dynamic distance; Step 4: After concatenating the multimodal features, the multimodal feature x is obtained and re-input into the multimodal graph neural network for training to obtain the multimodal fused context-dependent sentiment feature x'; Step 5: Jointly train the three unimodal graph neural networks and the multimodal graph neural network to obtain the optimal model parameters. The implementation of the graph neural network includes optional graph convolutional networks (GCNs) and graph attention networks (GATs). Step 6: Input the emotional video sample to be identified into the pre-trained feature extraction network, and then input it into the trained unimodal graph neural network and multimodal graph neural network to identify the emotional category of the speech sample.

2. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 1, obtain the video sample and transcribed text sample of the dataset, and extract the text, speech, and visual modal information from them as t, a, and v respectively. The specific steps include the following: Step 11: Obtain a transcribed text sample and assemble it into a transcription file in the format of "speaker ID-text-gender mapping-emotion label", where the speaker ID contains "conversation ID-speaker ID" and the gender mapping is a specific number / letter. Obtain the corresponding text sample t; Step 12: Corresponding to the transcribed text sample, extract the speech sample a corresponding to the dataset; Step 13: Corresponding to the transcribed text samples, extract the corresponding video image samples of the dataset, and extract video frames as visual samples v at specific intervals.

3. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 2, the pre-trained BERT, wav2vec2, and MTCNN models are used to extract features from text, speech, and visual samples, respectively, to obtain the sentiment features t', a', and v', respectively. The specific steps include the following: Step 21: For the extracted text sample, use the pre-trained BERT model to extract text features t′, and divide the training set and test set according to the dialogue level based on a fixed partitioning protocol; Step 22: For the extracted speech sample, use the pre-trained Wav2Vec2 model to extract the speech feature a' and align the text to divide the dataset; Step 23: For the extracted visual samples, use the pre-trained MTCNN model to extract the bounding box of the face using the MTCNN face detection algorithm, extract the facial action unit through the facial action coding system, extract the visual feature v', and align the text to divide the dataset.

4. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 3, the three modal sentiment features are input into the graph neural network, and single-modal training is performed on their contextual dependencies to further extract text modal features t", speech modal features a", and visual modal information v", which are integrated with the contextual dynamic distance. The specific steps include: Step 31: Based on the number of conversations M, construct a conversation sentence mask and align the structural information of the following figures in different batches to facilitate subsequent unified processing; Step 32: Encode the feature sequence through the graph neural network (GNN) and capture the contextual dependencies in the conversation. Each sentence in each conversation is treated as a node, and the initial node representation u of the sentence is generated. Step 33: Using the node representation u, a graph neural network containing conversation context information is constructed. This is presented as an adjacency matrix adj and edge index information edge index, which includes node information, node connectivity, and edge types. The window size is variable to capture key conversation information. Step 34: Output unimodal emotional features t", a", v".

5. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 4, the multimodal features are concatenated to obtain the multimodal feature x, which is then re-input into the multimodal graph neural network for training to obtain the multimodal fused context-dependent sentiment feature x'. The specific steps include: Step 41: Set the unimodal graph neural network to inference mode, perform inference on the original sample, and obtain unimodal features; Step 42: Concatenate the unimodal features in the order of "text-speech-video" to obtain the multimodal feature x; Step 43: Encode the feature sequence through the graph neural network (GNN) and repeat the single-modal network operation to capture the contextual dependencies in the conversation. Each sentence in each conversation is used as a node to generate the initial node representation u' of the sentence. Step 44: Using the node representation u', a graph neural network is constructed that contains the conversation context information. This is presented as an adjacency matrix adj' and edge index information edge index', which includes node information, node connectivity, and adjacent edge types. The window size is also variable to capture key conversation information. Step 45: Output the node multimodal emotional feature x' and output the prediction score for each category. In the emotion classification task, use the classification module to perform multi-class emotion classification prediction on the node feature x' and output the final emotion classification result. Step 46: The classification module directly selects the category with the largest node prediction score as the emotion classification result.

6. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network uses the graph convolutional network GCN. The specific operations of a single GCN layer are as follows: input unimodal samples, construct the sample matrix input, perform linear transformation according to the edge weight, and the edge weight is iterated through learning; multiply the adjacency matrix adj by the linearly transformed feature matrix to aggregate the feature information of each node and its neighboring nodes to realize the propagation and sharing of information in the graph structure; add the bias vector bias to the output, perform translation operation on the output features, and increase the expressive power of the model.

7. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 6 is characterized in that: The graph neural network uses the graph convolutional network (GCN). The specific steps of the graph convolutional network GCN are: input feature matrix, perform graph convolution operation, and obtain first-level features; the first-level features are output through the activation function, and added to the original output through residual connection as the input of the next-level graph convolution operation; the "GCN-ReLU-residual connection" is repeatedly constructed into a five-layer network architecture; the output node single-modal / multi-modal sentiment features.

8. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 1 is characterized in that: In step 5, the three unimodal graph neural networks and the multimodal graph neural network are jointly trained to obtain the optimal model parameters. The graph neural network uses a graph attention network (GAT). The specific operations of a single GAT layer are as follows: a unimodal sample is input, and the node feature matrix x and edge index matrix edge index of the sample are constructed. The edge connectivity is defined by the graph structure. Each edge uses the attention mechanism to calculate the attention coefficient relative to its neighboring nodes and perform a weighted sum of the neighboring node features to achieve adaptive information propagation. The attention weights are learned through a parameterized feedforward network to support multi-head attention mechanisms. Optionally, the outputs of multiple heads are concatenated or averaged. Dropout is applied to the input and attention weights to improve model generalization.

9. The multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network according to claim 8, characterized in that: The graph neural network uses the graph attention network (GAT). The GAT model is built as follows: input the feature matrix, perform the first-layer multi-head graph attention convolution, and obtain the first-level node features; the first-level features are processed by the activation function ELU and input into the second-layer graph attention convolution; the second-layer output is activated by ELU and dropout, and then sent to the output graph attention layer to output the final classification features; The last layer uses single-head attention output with the same dimension as the number of categories; it outputs the unimodal / multimodal sentiment features of the node.

10. A system based on the multimodal conversation emotion recognition method based on dynamic context dependency embedding graph neural network as claimed in claim 1, characterized in that: include: Feature extraction module, used to extract unimodal sample features of text, speech, and vision; A unimodal graph neural network module is used to train unimodal sample features and capture the dynamic distance dependencies of sentence contexts; Feature splicing module, used to perform feature splicing on single-modal training features to achieve multi-modal feature fusion; Multimodal graph neural network module, used to train multimodal fusion features and capture the dynamic distance dependencies of multimodal feature contexts; The sentiment classification module is used to perform sentiment discrimination and classification on the fused multimodal features.

Images