Incomplete multi-mode dialogue emotion recognition method and system based on speaker and time sequence information joint graph network

Abstract

The invention discloses an incomplete multi-mode dialogue emotion recognition method and system based on a speaker and time sequence information joint graph network, and the method comprises the steps: obtaining the deep features of a text mode, a voice mode and a visual mode in a dialogue through a feature extraction module, and guaranteeing the high expression capability of the features through a pre-training model; the random mode missing simulation module effectively simulates the data incomplete condition in a real scene, and the robustness of the model is improved; a bi-directional long-short term memory network (Bi-LSTM) is combined with a time sequence diagram network (TGNN) to capture context and time dynamic characteristics of a dialogue, and meanwhile, an interaction relationship between speakers is modeled through a speaker influence matrix, so that joint modeling of a time sequence and speaker information is realized; the deep features are further extracted through the graph convolutional network, and the emotion discrimination of the features is enhanced; finally, the modal reconstruction and emotion classification module significantly improves the accuracy and robustness of incomplete multi-modal dialogue emotion recognition through reconstruction of missing modals and multi-class emotion prediction, and is suitable for man-machine interaction and emotion analysis application in a complex real scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and in particular to an incomplete multi-modal conversation emotion recognition method and system based on a speaker and timing information joint graph network. BACKGROUND

[0002] Emotion Recognition in Conversation (ERC) is one of the core technologies to realize human-computer conversation, aiming to accurately identify the emotional state (such as happiness, sadness, anger, etc.) of participants by analyzing the semantic features in the conversation, the context relationship and the interaction mode of the conversation. With the rapid development of artificial intelligence technology, human-computer conversation systems have shown important value in the fields of emotional companionship, psychological counseling, etc. The research of this task not only involves semantic understanding in the field of natural language processing (NLP), but also needs to integrate the emotional computing model of psychology, and in recent years it has also received extensive attention from the academic community.

[0003] Due to the complexity of emotion recognition, a single modality cannot meet the demand, and text, vision and speech are the three modes of machine perception of human emotion, so a specific emotion can be conveyed through the three ways of expressing emotion, i.e. multi-modal conversation emotion recognition. However, in real-world environments, many factors can cause modalities to be missing. For example, speech may be missing due to background noise or sensor failure; text may not be available due to automatic speech recognition errors or unknown words; and faces may not be detected due to light, motion or occlusion. The problem of incomplete modalities increases the difficulty of accurately understanding the conversation, so learning from incomplete multi-modal data is a fundamental research area in machine learning.

[0004] Since the dependency relationship between speakers and different sentences needs to be considered in the conversation, and this relationship is non-Euclidean, the use of graph neural networks can achieve more effective and accurate modeling. Traditional GCNet models the speaker and the time independently, ignoring the relevance of the speaker's identity and the time, resulting in a jump or inconsistency in the recognition result. How to utilize the correlation information of the speaker's identity and the time, and fuse it with the timing emotional information of the conversation to achieve joint modeling is the key to improving the accuracy and robustness of conversation emotion recognition. SUMMARY

[0005] The present application provides an incomplete multi-modal conversation emotion recognition method and system based on a speaker and timing information joint graph network, which realizes joint modeling of speaker influence information and timing information, has good generalization and high recognition rate.

[0006] Technical scheme: The incomplete multi-modal conversation emotion recognition method based on a speaker and timing information joint graph network provided by the present application comprises the following steps:

[0007] Step 1, obtain dialogue information text, voice, and visual data in three modalities, respectively extract features using pre-trained models, and splice them into multi-modal features u of dialogue information;

[0008] Step 2, randomly discard part of the modal features to obtain modal missing features u';

[0009] Step 3, encode the modal missing feature sequence through a bidirectional long short-term memory network Bi-LSTM, and capture the context dependence in the dialogue, each sentence of the dialogue as a node, to generate an initial node representation u";

[0010] Step 4, use the node representation u" to construct a graph network TGNN containing dialogue timing information, which is presented in the form of an adjacency matrix T, containing node information, node connection relationship and adjacent edge type, and the window size is variable, which can capture key dialogue information;

[0011] Step 5, use the node representation u" to construct a matrix S containing speaker influence information, with a dimension of the number of speakers, and use the information contained in S to expand the matrix S into a matrix S' corresponding to TGNN containing speaker influence information, the node number is the same as the adjacency matrix of TGNN, and the corresponding node information is the weight of the speaker influence;

[0012] Step 6, fuse TGNN and the matrix containing speaker influence information, i.e. multiply the corresponding nodes to realize joint modeling of timing information and speaker influence, and then perform graph convolution on the TGNN containing speaker influence information to output the fused features u"';

[0013] Step 7, splice the features u'" with the original node representation u" and input them into Bi-LSTM to obtain the features q;

[0014] Step 8, use q to complete the reconstruction of the missing modal information and the emotion classification, and output the classification result.

[0015] Further, in step 1, obtaining dialogue information text, voice, and visual data in three modalities, respectively extracting features using pre-trained models, and splicing them into multi-modal features u of dialogue information includes the following steps:

[0016] Step 11, for the text modal information of the dialogue, use the pre-trained DeBERTa model to encode the semantic features;

[0017] Step 12, for the voice modal information of the dialogue, use the Wav2vec model to extract acoustic features for each sentence;

[0018] Step 13, for the visual modality information of the dialogue, use the pre-trained MA-Net as the visual feature extractor, use the MTCNN face detection algorithm to extract the aligned face, and then use the pre-trained MA-Net3 for face feature extraction;

[0019] Step 14, concatenate the extracted features in the three modalities to obtain the multi-modal feature u of the dialogue information.

[0020] Further, in step 2, a part of the modalities is randomly discarded to obtain the feature u' with missing modalities, which specifically includes the following steps:

[0021] Step 21, for each dialogue sample, define a modality missing mask, which is a Boolean vector with a length of M (the number of modalities), used to indicate the retention and discard of modalities in the current sample, wherein it is constrained that at least one modality is retained;

[0022] Step 22, based on the set missing probability p, generate a missing mask for each modality to randomly determine whether it is discarded. If all modalities are marked as missing, randomly select one modality to force retention to ensure that at least one modality feature exists;

[0023] Step 23, according to the generated modality missing mask, replace the modality features marked as missing with all zero values to simulate the case of missing modalities in the real scene, and finally obtain the multi-modal feature u' after missing modalities, as the input of the dialogue emotion recognition and modality reconstruction task.

[0024] Further, in step 3, encode the feature sequence with missing modalities through a bidirectional long short-term memory network Bi-LSTM, and capture the context dependence in the dialogue, generate an initial node representation u" for each sentence of the dialogue as a node, which specifically includes the following steps:

[0025] Step 31, organize the multi-modal feature sequence u' after missing modalities according to the dialogue turn (sentence level), and concatenate the multi-modal features of each sentence as the input feature sequence of the turn;

[0026] Step 32, input the above input sequence into a bidirectional long short-term memory network to capture the context information of the past and future of the dialogue respectively, and obtain the bidirectional context encoding feature of each sentence;

[0027] Step 33, use the bidirectional context encoding vector of each sentence as the corresponding graph node representation feature u" for subsequent joint graph network modeling based on speaker and timing information.

[0028] Further, in step 4, use the node representation u" to construct a graph network TGNN containing dialogue timing information, which specifically includes the following steps:

[0029] Step 41, based on the node representation feature u" of each sentence obtained in step 3, the graph structure corresponding to the dialogue turn is constructed, in which the node represents the corresponding speaking sentence, and the edge represents the temporal relationship between the sentences.

[0030] Step 42, define the adjacency matrix T of TGNN, use the window to capture the local context information, based on the center node in the window, establish the edge connection between the related nodes, and according to the connection relationship of each edge, give the time sequence relationship label W i , to capture the time dynamic characteristics in the dialogue.

[0031] Further, in step 5, the node representation u" is used to construct a matrix S containing speaker influence information, which includes the following steps:

[0032] Step 51, input the utterance node feature u" obtained in step 3, calculate the element information of the speaker influence matrix S according to the relationship between the speakers, the calculation principle is: define the speakers A and B, the influence of speaker B on speaker A is the mean value of all utterance feature tensors said by speaker A and the previous sentence said by B, denoted as s 21 ;

[0033] Step 52, widen the dimension of matrix S to the dimension of the adjacency matrix of TGNN, form a global speaker influence matrix S', each node of which corresponds to the speaker influence before and after the node belongs to in TGNN. The matrix S' has no adjacency relationship, and only represents the speaker influence before and after.

[0034] Further, in step 6, the TGNN is fused with the matrix containing the speaker influence information, that is, the corresponding nodes are multiplied, realizing the joint modeling of the time sequence information and the speaker influence, and then the TGNN with the fused speaker influence information is subjected to graph convolution, outputting the fused feature u'", which includes the following steps:

[0035] Step 61, fuse the TGNN adjacency matrix constructed in step 4 and the global speaker influence matrix S' obtained in step 5 at the feature level, that is, multiply the nodes of the TGNN adjacency matrix by the corresponding elements of S', to obtain the TGNN with fused speaker influence information, which is used to encode the time dynamic characteristics of the dialogue and the speaker influence information before and after at the same time;

[0036] Step 62, based on the fused graph representation, use the graph convolution network to propagate and update the features of the nodes in the graph, output the fused node feature u'" as the input of the subsequent emotion classification and reconstruction task.

[0037] Further, in step 7, the feature u'" is spliced with the original node representation u", and input into the Bi-LSTM to obtain the feature q, which includes the following steps:

[0038] Step 71, the fusion feature u'" obtained by graph convolution in step 6 is spliced with the node representation u" generated by Bi-LSTM encoding in step 3 in the feature dimension to form a joint feature representation;

[0039] Step 72, the spliced joint feature representation is input into a Bi-LSTM layer to model deeper context dependencies between nodes through bidirectional information flow, and output updated node features q.

[0040] Further, in step 8, q is used to complete the final reconstruction of missing modal information and emotion classification, and the classification results are output, including the following steps:

[0041] Step 81, the node features q are input into two decoding modules respectively, one of which is used to predict and reconstruct the missing modal information, and the other is used to predict the emotion category label of the current node;

[0042] Step 82, in the missing modal reconstruction task, the reconstructed module is used to recover the randomly discarded modal features, and the reconstruction error between the original non-missing modal features and the recovered modal features is calculated;

[0043] Step 83, in the emotion classification task, the classification module is used to perform multi-class emotion classification prediction on the node features q, and output the final emotion classification results.

[0044] Correspondingly, an incomplete multi-modal dialogue emotion recognition system based on speaker and time sequence information joint graph network includes: a feature extraction module for extracting information in text, speech, and visual modalities in dialogue information;

[0045] A modal missing simulation module artificially creates missing modal information to simulate the missing modal information that may be encountered in real situations;

[0046] A sentence-level feature encoding module uses Bi-LSTM to encode the feature sequence after modal loss to generate sentence-level node features;

[0047] A time sequence graph network construction module is used to construct a graph network based on time sequence relationships, and model the time sequence information in the dialogue based on adjacency matrix information;

[0048] A speaker influence modeling module is used to construct a matrix containing speaker influence information before and after, and expand it to the same dimension as the time sequence graph;

[0049] A time sequence and speaker information fusion module is used to multiply the time sequence features and speaker features in the graph network to jointly model the time sequence information and speaker influence information, and perform graph convolution on the fused graph network to extract deep features in the graph network;

[0050] a feature re-encoding module, configured to splice and fuse the original sentence-level features and the features extracted by the graph convolution module, input the features into a Bi-LSTM again, and obtain re-encoded features;

[0051] a modal reconstruction and emotion classification module, configured to reconstruct the missing modal features, and predict the category of the emotion based on the re-encoded features.

[0052] Advantages: Compared with the prior art, the present application has the following remarkable advantages: the present application obtains deep features of dialogue text, speech and vision of three modalities through a feature extraction module, uses a pre-trained model (such as DeBERTa, Wav2vec and MA-Net) to ensure high expression capability of the features, and uses a random modal missing simulation module to effectively simulate the incomplete data in a real scene, thereby improving the robustness of the model; a bidirectional long short-term memory network (Bi-LSTM) combined with a time graph network (TGNN) captures the context and time dynamic characteristics of the dialogue, and at the same time models the interaction relationship between speakers through a speaker influence matrix, thereby realizing joint modeling of time sequence and speaker information; a graph convolution network further extracts deep features, thereby enhancing the emotion discriminability of the features; finally, a modal reconstruction and emotion classification module reconstructs the missing modal and predicts multiple emotions, thereby significantly improving the accuracy and robustness of incomplete multi-modal dialogue emotion recognition, and being suitable for human-computer interaction and emotion analysis applications in complex real scenes. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a method flowchart of the present application.

[0054] Figure 2 is a comparison diagram of experimental results of the present application and other methods in four-classification tasks and six-classification tasks under the IEMOCAP data set. DETAILED DESCRIPTION

[0055] As shown in Figure 1 , a method for incomplete multi-modal dialogue emotion recognition based on a speaker and time sequence information joint graph network comprises the following steps:

[0056] (1) Obtain data in dialogue information text, speech and vision of three modalities, extract features by using a pre-trained model respectively, and splice the features into multi-modal features u of the dialogue information.

[0057] Specifically, it includes: (1-1) for the text modality information of the dialogue, using a pre-trained DeBERTa model to encode semantic features; (1-2) for the speech modality information of the dialogue, using a Wav2vec model to extract acoustic features for each sentence; (1-3) for the visual modality information of the dialogue, using a pre-trained MA-Net as a visual feature extractor, extracting an aligned face using an MTCNN face detection algorithm, and then using a pre-trained MA-Net3 to extract facial features; (1-4) concatenating the extracted features in the three modalities to obtain multi-modal features u of the dialogue information.

[0058] In this embodiment, the semantic feature dimension extracted based on the IEMOCAP database is 1024, the acoustic feature dimension extracted is 512, and the visual feature dimension extracted is 1024.

[0059] (2) Randomly discard part of the modal features to obtain modal missing features u'.

[0060] Specifically, it includes: (2-1) for each dialogue sample, defining a modal missing mask, which is a Boolean vector with a length of M (the number of modalities) and is used to indicate the retention and discard of the modalities in the current sample, wherein at least one modality is retained; (2-2) based on the set missing probability p, generating a missing mask for each modality to randomly determine whether it is discarded. If all modalities are marked as missing, randomly select one modality to force retention to ensure that at least one modality feature exists; (2-3) according to the generated modal missing mask, replace the modal features marked as missing with all zero values to simulate the case of modal missing in the real scene, and finally obtain the multi-modal features u' after modal missing, as the input of the dialogue emotion recognition and modality reconstruction task.

[0061] In this embodiment, the number of modalities is 3, the set missing probability is 0.2, and the dimension of the concatenated multi-modal features is 2560.

[0062] (3) Encode the modal missing feature sequence through a bidirectional long short-term memory network (Bi-LSTM) and capture the context dependence in the dialogue, each sentence of the dialogue as a node, to generate an initial node representation u".

[0063] Specifically, it includes: (3-1) organizing the multi-modal feature sequence u' after the modal loss according to the dialogue turn (sentence level), and splicing the multi-modal feature of each sentence as the input feature sequence of the turn; (3-2) inputting the above input sequence into a bidirectional long short-term memory network to capture the context information of the dialogue history and future from the front and back respectively, and obtaining the bidirectional context encoding feature of each sentence; (3-3) taking the bidirectional context encoding feature of each sentence as the corresponding node representation feature u", which is used for subsequent joint graph network modeling based on speaker and timing information.

[0064] In this embodiment, the number of hidden nodes of Bi-LSTM is 200, which can be changed by parameter adjustment.

[0065] (4) Using the node representation feature u", a graph network TGNN containing dialogue timing information is constructed, which is presented in the form of an adjacency matrix T, containing node information, connection relationship of nodes and type of adjacent edges, etc.

[0066] Specifically, it includes: (4-1) based on the node representation feature u" of each sentence obtained in step (3), constructing a graph structure corresponding to the dialogue turn, in which the node represents the corresponding speaking sentence, and the edge represents the timing relationship between the sentences; (4-2) defining the adjacency matrix T of TGNN, using a window that captures local context information, based on the center node in the window, establishing edge connection between related nodes, and according to the connection relationship of each edge, assigning a timing relationship label W i to capture the time dynamic characteristics in the dialogue.

[0067] In this embodiment, the size of the window is 2, that is, the center of the window and the first two points and the last two points have adjacent relationships; the type of TGNN adjacent edge is divided into past, present and future according to the timing relationship.

[0068] (5) Using the node representation feature u", a matrix S containing speaker before and after influence information is constructed, with a dimension of the number of speakers. Using the information contained in S, the matrix S is widened to a matrix S' corresponding to TGNN containing speaker before and after influence information, the number of nodes is the same as the adjacency matrix of TGNN, and the corresponding node information is the weight of the speaker influence.

[0069] Specifically, it includes: (5-1) inputting the utterance node feature u" obtained in step (3), calculating the element information of the speaker influence matrix S according to the relationship before and after the speaker, and the calculation principle is: defining speakers A and B, the influence of speaker B on speaker A is the mean value of all utterance feature tensors said by speaker A before and after the previous sentence said by speaker B, denoted as s 21, the calculation method of the remaining elements is the same; (5-2) the dimension of the matrix S is widened to the dimension of the adjacency matrix of the TGNN to form a global speaker influence matrix S', each node of which corresponds to the speaker influence before and after the node belongs to in the TGNN. In particular, the matrix S' has no adjacency relationship and only represents the speaker influence before and after.

[0070] In this embodiment, the dialogue information in the IEMOCAP dataset is all in a two-person scene, so the speaker influence matrix is a 2-dimensional square matrix, that is, the dimension is the same as the number of speakers; the dimension of the widened speaker influence matrix is the same as the number of points of the TGNN, that is, the same as the dialogue length in a batch.

[0071] (6) Fuse the TGNN and the matrix containing the speaker influence information, that is, multiply the corresponding nodes to realize the joint modeling of the temporal information and the speaker influence. Then perform graph convolution on the TGNN with the speaker influence information fused to output the fused feature u'''.

[0072] Specifically, it includes: (6-1) fuse the TGNN adjacency matrix constructed in step (4) and the global speaker influence matrix S' obtained in step (5) at the feature level, that is, multiply the nodes of the TGNN adjacency matrix by the corresponding elements of S' to obtain the TGNN with the speaker influence information fused, which is used to encode the temporal dynamic characteristics of the dialogue and the speaker influence information before and after the dialogue; (6-2) based on the fused graph representation, use the graph convolution network to propagate and update the features of the nodes in the graph, and output the fused node feature u''' as the input of the subsequent emotion classification and reconstruction task.

[0073] In this embodiment, the operation of the graph convolution network is based on the relational graph convolution layer and the standard graph convolution layer, and the convolution layer dimension is 64.

[0074] (7) Concatenate the feature u''' and the original node representation feature u'' to input to obtain the feature q.

[0075] Specifically, it includes: (7-1) concatenate the fused feature u''' obtained by graph convolution in step (6) and the node representation feature u'' generated by Bi-LSTM encoding in step (3) in the feature dimension to form a joint feature representation; (7-2) input the concatenated joint feature representation into the Bi-LSTM layer to model the deeper context dependency between nodes through bidirectional information flow, and output the updated node feature q.

[0076] In this embodiment, the hidden dimension of the LSTM layer is 600.

[0077] (8) Use q to complete the final reconstruction of the missing modal information and the emotion classification, and output the classification result.

[0078] Specifically, the method comprises: (8-1) inputting the node features q into two decoding modules respectively, one of which is used to predict and reconstruct the information of the missing modality, and the other is used to predict the emotional category label of the current node; (8-2) in the missing modality reconstruction task, the missing modality features are recovered through the reconstruction module, and the reconstruction error between the original non-missing modality features and the recovered features is calculated; (8-3) in the emotion classification task, the classification module is used to perform multi-class emotion classification prediction on the node features q, and finally output the emotion classification result.

[0079] In the embodiment, the reconstruction module maps the features to a vector with the same total dimension as the original input through a linear fully connected layer, which is used to reconstruct the features; and the emotion classification module maps the features to the logits of the number of emotion categories through a linear fully connected layer. For the four-class task based on IEMOCAP, the emotion category is 4, and for the six-class task, the emotion category is 6.

[0080] To verify the effectiveness and necessity of the time sequence emotion information modeling network, experiments are performed on the IEMOCAP and the data set. The experimental protocol is five-fold cross-validation of leave-one-session for the IEMOCAP data set (the IEMOCAP data set contains five sessions), and the four-class and six-class emotion classification tasks are performed. The F1-Score (weighted F1 score) is selected as the evaluation index, and the results are shown in Table 1. Figure 2 As can be observed, the application achieves a higher F1-Score on the two tasks based on the IEMOCAP data set.

[0081] The embodiment also provides an incomplete multi-modal dialogue emotion recognition system based on joint modeling of speaker identity and time information, comprising:

[0082] The feature extraction module is configured to extract information in three modalities of text, speech and vision in the dialogue information.

[0083] The modality missing simulation module is configured to artificially manufacture missing of modality information, and simulate the missing of modality information that may be encountered in a real scene.

[0084] The sentence-level feature encoding module is configured to use Bi-LSTM to encode the feature sequence after the missing of the modalities, and generate sentence-level node features.

[0085] The time sequence graph network construction module is configured to construct a graph network based on a time sequence relationship, and model the time sequence information in the dialogue based on adjacency matrix information.

[0086] The speaker influence modeling module is configured to construct a matrix containing speaker influence information before and after, and expand the matrix to the same dimension as the time sequence graph.

[0087] The time sequence and speaker information fusion module is configured to multiply the time sequence features and the speaker features in a graph network, to realize joint modeling of time sequence information and speaker influence information, and to perform graph convolution on the fused graph network to extract deep features in the graph network.

[0088] The feature re-encoding module is configured to splice and fuse the original sentence-level features and the features extracted by the graph convolution module, input the spliced and fused features into a Bi-LSTM again, and obtain re-encoded features.

[0089] The modality reconstruction and emotion classification module is configured to reconstruct missing modality features, and predict the category of the emotion based on the re-encoded features.

[0090] The feature extraction module specifically comprises: a text feature extraction unit configured to extract deep semantic features from input text based on a DeBERTa pre-training language model; an audio feature extraction unit configured to extract audio features from input audio based on a Wav2Vec pre-training model; a visual feature extraction unit configured to extract visual features from input video frames based on a MA-Net pre-training model; and a feature splicing unit configured to splice the text, audio and visual features into complete multi-modal features.

[0091] The modality missing simulation module specifically comprises: a mask generation unit configured to randomly generate a modality missing mask to control random loss of modality information and retain at least one kind of modality information; and a missing application unit configured to apply the mask to the spliced multi-modal features to generate missing multi-modal features.

[0092] The sentence-level feature encoding module specifically comprises a bidirectional long short-term memory network configured to encode the context-dependent features of the sentence features after the loss of modalities to generate sentence-level node features.

[0093] The time sequence graph network construction module specifically comprises a time sequence adjacency matrix generation unit configured to establish an adjacency relationship matrix between nodes according to the order of sentences in the dialogue and a time window. The window parameter is adjustable to control the connection range and time sequence dependence strength of adjacent nodes.

[0094] The speaker influence modeling module specifically comprises a speaker adjacency matrix generation unit configured to establish a speaker influence matrix according to the identity of the speaker.

[0095] The time sequence and speaker information fusion module specifically comprises: an information fusion unit configured to generate time sequence feature graphs fused with speaker information; a graph convolution network comprising a relationship graph convolution layer and a standard graph convolution layer, wherein the relationship graph convolution layer can extract global time sequence information based on different time sequence relationships of adjacent edges; and an activation function configured to improve the nonlinear expression ability of the model.

[0096] The feature re-encoding module specifically comprises: a feature splicing unit, configured to splice the features extracted by the graph convolution and the original node features into comprehensive features. A bidirectional long short-term memory network is configured to perform re-contextualization on the comprehensive feature sequence to generate re-encoded features.

[0097] The modality reconstruction and emotion classification module specifically comprises: a modality reconstruction unit, configured to reconstruct the information of the missing modality from the re-encoded features; and an emotion classification unit, configured to predict the emotion category from the re-encoded features.

Claims

1. A method for emotion recognition in incomplete multimodal conversations based on a joint graph network of speaker and temporal information, characterized by: The steps include: Step 1: Obtain conversation information data in three modalities: text, speech, and vision. Use pre-trained models to extract features from each modality and combine them into a multimodal feature u of the conversation information. Step 2: Randomly discard the features of some modes to obtain the missing features u'; Step 3: Use a bidirectional long short-term memory (Bi-LSTM) network to encode the missing modality feature sequence and capture the contextual dependencies in the conversation. Each sentence of the conversation is treated as a node to generate the initial node representation u". Step 4: Use the node representation u" to construct a graph network (TGNN) containing conversation timing information. This is presented as an adjacency matrix T, which includes node information, node connectivity, and edge types. The window size is variable to capture key conversation information. Step 5: Use the node representation u' to construct a matrix S containing the speaker's influence information. The dimension is the number of speakers. Using the information contained in S, the matrix S is widened to a matrix S' containing the speaker's influence information corresponding to the TGNN. The number of nodes is the same as the adjacency matrix of the TGNN, and the corresponding node information is the weight of the speaker's influence. Step 6: Fuse the TGNN with the matrix containing the speaker influence information, that is, multiply the corresponding nodes to achieve joint modeling of time series information and speaker influence. Then perform graph convolution on the TGNN fused with the speaker influence information to output the fused feature u''; Step 7: Concatenate the feature u'' with the original node representation u'' and input it into Bi-LSTM to obtain the feature q; Step 8: Use q to complete the final reconstruction of the missing modal information and emotion classification, and output the classification results.

2. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 1, we obtain conversation information in three modalities: text, speech, and vision. We use pre-trained models to extract features from each modality and combine them into a multimodal feature u of the conversation information. The specific steps are as follows: Step 11: For the text modal information of the conversation, use the pre-trained DeBERTa model to encode semantic features; Step 12: For the speech modality information of the conversation, use the Wav2vec model to extract acoustic features for each sentence; Step 13: For the visual modality information of the conversation, use the pre-trained MA-Net as the visual feature extractor, use the MTCNN face detection algorithm to extract the aligned faces, and then use the pre-trained MA-Net3 to extract facial features; Step 14: Concatenate the extracted features under the three modalities to obtain the multimodal feature u of the conversation information.

3. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 2, the features of some modes are randomly discarded to obtain the missing features u′ of the modes. The specific steps include: Step 21: For each conversation sample, define a modality loss mask, which is a Boolean vector of length M (number of modalities) to indicate whether the modalities are retained or discarded in the current sample, with the constraint that at least one modality is retained; Step 22: Based on the set missing probability p, a missing mask is generated for each modality, and it is randomly determined whether it is discarded. If all modalities are marked as missing, a modality is randomly selected and forced to be retained to ensure that at least one modality feature exists; Step 23: Based on the generated modality loss mask, replace the modality features marked as missing with all zero values ​​to simulate the modality loss situation in real scenarios. Finally, the multimodal feature u' after modality loss is obtained as the input for the dialogue emotion recognition and modality reconstruction tasks.

4. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 3, a bidirectional long short-term memory (Bi-LSTM) network is used to encode the missing modality feature sequence and capture the contextual dependencies in the conversation. Each sentence of the conversation is treated as a node, and the initial node representation u is generated. The specific steps are as follows: Step 31: Organize the multimodal feature sequence u' after modality loss by conversation turn (sentence level), and concatenate the multimodal features of each sentence into the input feature sequence of that turn; Step 32: Feed the above input sequence into a bidirectional long short-term memory network to capture the conversation history and future context information from the forward and backward directions, respectively, to obtain bidirectional context encoding features for each sentence; Step 33: Use the bidirectional context encoding vector of each sentence as the corresponding graph node representation feature u' for subsequent joint graph network modeling based on speaker and timing information.

5. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 4, using the node representation u", constructing a graph network TGNN containing conversation timing information specifically includes the following steps: Step 41: Based on the node representation feature u' of each sentence obtained in step 3, a graph structure corresponding to the conversation turn is constructed. In the graph structure, nodes represent corresponding utterances, and edges represent the temporal relationship between sentences. Step 42: Define the adjacency matrix T of TGNN, use the window that captures local context information, establish edge connections between related nodes based on the central node in the window, and assign a temporal relationship label W based on the connection relationship of each edge. i , to capture the temporal dynamics in conversations.

6. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 5, the node representation u' is used to construct a matrix S containing the speaker's influence information before and after. Specifically, the following steps are included: Step 51: Input the utterance node feature u' obtained in step 3 and calculate the element information of the speaker influence matrix S according to the relationship between the previous and next speakers. The calculation principle is: define speakers A and B, and the influence of speaker B on speaker A is the mean of all utterance feature tensors of the previous sentence spoken by speaker A, which is recorded as s 21 ; Step 52: Expand the dimension of the matrix S to the dimension of the adjacency matrix of the TGNN to form a global speaker influence matrix S', in which each node corresponds to the speaker's before and after influence to which the node belongs in the TGNN. The matrix S' has no adjacency relationship and only represents the speaker's before and after influence.

7. The method for emotion recognition in incomplete multimodal conversations based on a joint graph network of speaker and temporal information according to claim 1, wherein: In step 6, the TGNN is fused with the matrix containing the speaker influence information, that is, the corresponding nodes are multiplied to achieve joint modeling of time series information and speaker influence. Then, the TGNN fused with the speaker influence information is subjected to graph convolution to output the fused feature u''. The specific steps include the following: Step 61: Fuse the TGNN adjacency matrix constructed in step 4 with the global speaker influence matrix S' obtained in step 5 at the feature level. That is, multiply the nodes of the TGNN adjacency matrix by the corresponding elements of S' to obtain a TGNN that incorporates speaker influence information, thereby encoding both the temporal dynamics of the conversation and the contextual speaker influence information. Step 62: Based on the fused graph representation, use the graph convolutional network to propagate and update the features of the nodes in the graph, and output the fused node features u'' as the input for subsequent emotion classification and reconstruction tasks.

8. The incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information according to claim 1 is characterized in that: In step 7, the feature u'' is concatenated with the original node representation u'' and input into the Bi-LSTM to obtain the feature q. The specific steps include the following: Step 71: Concatenate the fusion feature u'' obtained by graph convolution in step 6 and the node representation u'' generated by Bi-LSTM encoding in step 3 in the feature dimension to form a joint feature representation; Step 72: Input the concatenated joint feature representation into the Bi-LSTM layer, model deeper contextual dependencies between nodes through bidirectional information flow, and output the updated node feature q.

9. The method for emotion recognition in incomplete multimodal conversations based on a joint graph network of speaker and temporal information according to claim 1, wherein: In step 8, q is used to complete the reconstruction of the final missing modal information and emotion classification, and the classification results are output. The specific steps include the following: Step 81: Input the node feature q into two decoding modules respectively, one of which is used to predict and reconstruct the missing modal information, and the other is used to predict the emotion category label of the current node; Step 82: In the missing modal reconstruction task, the randomly discarded modal features are restored through the reconstruction module, and the reconstruction error between the modal features and the original non-missing modal features is calculated; Step 83: In the emotion classification task, the classification module is used to perform multi-class emotion classification prediction on the node feature q and output the final emotion classification result.

10. A system based on the incomplete multimodal conversation emotion recognition method based on a joint graph network of speaker and temporal information as claimed in claim 1, characterized in that: include: Feature extraction module, used to extract information from the conversation information in three modalities: text, speech, and vision; The modal loss simulation module artificially creates the loss of modal information to simulate the modal information loss that may be encountered in real scenarios; The sentence-level feature encoding module uses Bi-LSTM to perform context encoding on the feature sequence after modality loss to generate sentence-level node features; The temporal graph network construction module is used to build a graph network based on temporal relationships and model the temporal information in the conversation based on the adjacency matrix information; The speaker influence modeling module is used to construct a matrix containing the speaker's influence information before and after, and expand it to the same dimension as the time series graph; The time series and speaker information fusion module is used to multiply time series features and speaker features in the graph network to achieve joint modeling of time series information and speaker influence information. At the same time, the graph network after fusion modeling is subjected to graph convolution to extract deep features in the graph network. The feature re-encoding module is used to concatenate and fuse the original sentence-level features with the features extracted by the graph convolution module, and then input them into the Bi-LSTM again to obtain the re-encoded features; The modality reconstruction and emotion classification module is used to reconstruct the missing modality features and predict the emotion category based on the re-encoded features.

Images