Speaker Relationship-Based Multimodal Dialogue Sentiment-Reason Pair Extraction Method

Abstract

This paper proposes a method for extracting emotion-cause pairs from multimodal conversations based on speaker relationships. The method includes: inputting multimodal conversations into a feature extractor and combining it with a speaker encoder for feature extraction; utilizing a multimodal graph convolutional network to perform feature interaction and fusion on the multimodal features; utilizing a feedforward network to classify and predict the final comprehensive feature representation; utilizing emotion- and cause-representation representations to create candidate emotion-cause pairs, utilizing a speaker relationship matrix to guide the extraction of candidate emotion-cause pairs, and utilizing an expert network and a gated network for classification and prediction to obtain predicted emotion-cause pairs. By innovatively constructing a speaker relationship matrix, this paper incorporates the dynamically changing social relationship network into the considerations for emotion-cause pair extraction, overcoming the limitation of existing methods that rely on simple dot product operations to evaluate the relevance of utterance pairs.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for extracting emotion-reason pairs from multimodal dialogues based on speaker relationships. Background Art

[0002] Emotion-Cause Pair Extraction (ECPE), a key natural language processing task, aims to identify emotional expressions and their triggering causes in text. This task is of great significance for enhancing the depth of sentiment analysis and generating empathetic responses, and therefore continues to attract widespread attention in the research community. Current research methods mainly focus on text content analysis, especially identifying emotions and causes at the sentence level. However, this single-text approach fails to fully utilize the rich multimodal contextual information in daily interactions. In real-world situations, emotions are often presented through a combination of textual and non-textual cues. Relying solely on textual modalities may lead to an incomplete understanding of emotions and their triggers, thereby restricting the overall effectiveness of the ECPE task.

[0003] Based on this understanding, the Multimodal Emotion-Cause Pair Extraction (MECPE) task was proposed. It aims to improve the extraction of emotion-cause pairs by integrating multimodal information. The MECPE task receives multimodal conversation input consisting of text, visuals, and audio. Its goal is to identify the sentiment and cause of all utterances and construct emotion-cause pairs (ECPs) at the utterance level. This task requires the system to deeply understand the multimodal input information and the conversational context, while accurately detecting sentiment, causes, and the relationship between them. This poses a significant challenge to accuracy.

[0004] For this task, existing research has proposed a two-stage approach that integrates multimodal features and utilizes an interaction matrix to detect emotions, causes, and their corresponding relationships. Although this approach has shown some effectiveness in practice, we observe two major limitations that may affect the overall performance of the model.

[0005] The primary problem is that most existing multimodal emotion-cause pair extraction models directly aggregate multimodal features to construct discourse-level representations and then perform cause extraction. This approach ignores the differentiated contributions of different modalities in triggering specific emotions.

[0006] Second, current methods use a simple dot product operation to evaluate the relevance between utterance pairs when extracting emotion-cause pairs. This mechanical approach has significant limitations. In conversational scenarios, in particular, the social relationships between speakers form a complex interactive network. These relationships significantly influence emotional expression and causal attribution, but existing methods fail to effectively account for this crucial dimension.

[0007] More importantly, these complex social relationship networks often change dynamically during conversations. A character may maintain different types of relationships with multiple other characters simultaneously, further increasing the complexity of extracting emotion-cause pairs. Relying solely on simple dot product operations to evaluate the association between utterance pairs clearly cannot fully capture this complex social relationship dynamics and its impact on the formation of emotion-cause pairs. Summary of the Invention

[0008] In view of the above situation, the main purpose of the present invention is to propose a multimodal dialogue emotion-reason pair extraction method based on speaker relations to solve the above technical problems.

[0009] The present invention proposes a method for extracting emotion-reason pairs from multimodal dialogues based on speaker relationships, the method comprising the following steps:

[0010] Step 1: Build a feature extractor based on the feature extraction mechanism. The feature extractor includes a pre-trained language model, a bidirectional long short-term memory network, an expanded three-dimensional convolutional neural network model, an embedding layer, and a speaker encoder. Build a graph convolutional network based on the feature interaction and fusion mechanism, a gated network based on the gating mechanism, and an expert network based on the expert mixture mechanism. The feature extractor, graph convolutional network, feedforward network, expert network, and gated network constitute the prediction model.

[0011] Step 2: Input the multimodal conversation into the feature extractor, and extract features of the multimodal conversation in combination with the speaker encoder to obtain multimodal features;

[0012] Step 3: Use the multimodal graph convolutional network to perform feature interaction and fusion on the multimodal features to obtain the final comprehensive feature representation;

[0013] Step 4: Use the feedforward network to classify and predict the final comprehensive feature representation, and obtain the emotional representation, emotional prediction results, cause representation, and cause prediction results respectively;

[0014] Step 5: Use the emotion representation and the cause representation to perform candidate emotion-cause pairing, use the speaker relationship matrix to guide the extraction of candidate emotion-cause pairings, and use the expert network and the gated network to perform classification prediction to obtain the prediction results of the emotion-cause pair;

[0015] Construct sentiment prediction loss based on sentiment prediction results;

[0016] Construct cause identification loss based on the prediction results of cause aspects;

[0017] Construct the emotion-reason pair classification loss based on the final emotion-reason pair;

[0018] The prediction model is optimized using sentiment prediction loss, cause identification loss, and sentiment-cause pair classification loss to obtain the optimized prediction model;

[0019] The multimodal dialogue is input into the optimized prediction model to obtain the final prediction result.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. By employing specialized feature extractors to extract features from three modalities, text, audio, and vision, this method avoids the limitations of existing methods that directly aggregate multimodal features. This allows for a more comprehensive capture of multimodal cues of emotional expression, leveraging the differentiated contributions of different modalities in triggering specific emotions. Experimental results show that compared to the BASE method, which relies solely on a single modality, MM-SRM achieves a 2.75% improvement in F1, demonstrating the importance of effective fusion of multimodal information for task performance.

[0022] 2. This invention innovatively constructs a speaker relationship matrix, incorporating dynamically changing social relationship networks into the considerations for emotion-cause pair extraction, overcoming the limitations of existing methods that use simple dot product operations to evaluate the relevance of discourse pairs. Through a learnable speaker relationship matrix and a relationship-based expert system, this method can more accurately capture the complex emotion-cause association patterns in conversations. Compared to the state-of-the-art HILO method, MM-SRM improves the F1 value by 0.78%, demonstrating that considering social relationship information between speakers has a positive effect on emotion-cause pair recognition.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of the steps of the multimodal dialogue emotion-reason pair extraction method based on speaker relations proposed by the present invention.

[0025] Figure 2 This is an architectural diagram of the multimodal emotion cause identification method based on the speaker relationship-based multimodal dialogue emotion-cause pair extraction method proposed in the present invention. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0027] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0028] See also Figure 1 The embodiment of the present invention proposes a method for extracting emotion-reason pairs from multimodal dialogues based on speaker relationships, the method comprising the following steps:

[0029] Step 1. Construct a feature extractor based on the feature extraction mechanism. The feature extractor includes a pre-trained language model, a bidirectional long short-term memory network, an expanded three-dimensional convolutional neural network model, an embedding layer, and a speaker encoder. Construct a graph convolutional network based on the feature interaction and fusion mechanism, a gated network based on the gating mechanism, and an expert network based on the expert mixture mechanism. The feature extractor, graph convolutional network, feedforward network, expert network, and gating network constitute the prediction model.

[0030] Step 2: Input the multimodal conversation into the feature extractor, and extract features of the multimodal conversation in combination with the speaker encoder to obtain multimodal features.

[0031] See also Figure 2 In step 2, the multimodal conversation is input into the feature extractor, and the speaker encoder is combined to extract features of the multimodal conversation to obtain multimodal features, which specifically includes the following steps:

[0032] Input a multimodal conversation, perform sequence concatenation on the multimodal conversation, and use markers to separate the input sequence. Input the input sequence into the pre-trained language model and perform feature extraction in combination with the speaker encoder to obtain the extracted text token vector representation;

[0033] Use an open source toolkit to extract features from multimodal conversations to obtain raw audio features. Input the raw audio features into a bidirectional long short-term memory network and combine them with a speaker encoder to extract features and obtain extracted audio features.

[0034] The dilated 3D convolutional neural network model is used to extract features from multimodal dialogues to obtain extracted raw visual features. The extracted raw visual features are then subjected to feature extraction using a bidirectional long short-term memory network combined with a speaker encoder to obtain an extracted visual representation.

[0035] The multimodal dialogue is represented by a vector to obtain the represented vector, which is passed through the embedding layer and mapped with the speaker encoder to obtain the speaker embedding vector.

[0036] Input multimodal discourse, the corresponding process has the following relationship:

[0037] ;

[0038] in, Represents a multimodal conversation, Indicates the first speaker, Indicates the first utterance, Indicates the A speaker, Indicates the a discourse;

[0039] In the steps of concatenating multimodal dialogue sequences, separating them using markers to obtain input sequences, feeding the input sequences into a pre-trained language model, and extracting features using a speaker encoder to obtain the extracted text token vector representation, the corresponding process has the following relationship:

[0040] ;

[0041] in, represents the entire sequence input to the pre-trained language model, Represents the start token in the pre-trained language model, Represents the end marker in the pre-trained language model, represents the first token in the first text utterance, represents the first token in the second text utterance, Indicates the first start tag The vector representation of Indicates the second start tag The vector representation of Indicates the end tag The vector representation of Indicates that it has been processed by the pre-trained language model. Indicates the start tag Vector representation of ;

[0042] It should be noted that Used to mark the beginning of a sentence, Contains all the words that are spliced ​​together, as well as special markers for separation. Used to mark the end of a sentence.

[0043] In the steps of extracting features from multimodal conversations using an open source toolkit to obtain extracted raw audio features, inputting the raw audio features into a bidirectional long short-term memory network, and combining them with a speaker encoder to extract features to obtain extracted audio features, the corresponding process has the following relationship:

[0044] ;

[0045] in, Indicates the The original audio features of the utterance, Indicates that the feature extraction is done by the open source toolkit. Indicates the The audio features of an utterance, Indicates that it has been processed by a bidirectional long short-term memory network;

[0046] It should be noted that It is an open source toolkit for audio feature extraction.

[0047] In the steps of extracting features from a multimodal conversation using an expanded 3D convolutional neural network model to obtain extracted raw visual features, and extracting features from the extracted raw visual features using a bidirectional long short-term memory network combined with a speaker encoder to obtain an extracted visual representation, the corresponding processes have the following relationship:

[0048] ;

[0049] in, Indicates the The original visual features of each utterance, It means that after the expansion of the three-dimensional convolutional neural network model, Indicates the visual features, Indicates feature extraction of original visual features;

[0050] It should be noted that A deep neural network model for video analysis and visual feature extraction. This model can effectively capture the spatiotemporal features in video sequences through a three-dimensional convolutional neural network architecture. Used to transform raw features into high-level feature representations with contextual semantic information.

[0051] In the steps of representing the multimodal dialogue row vector, obtaining the represented vector, passing the represented vector through the embedding layer, and performing a mapping operation in conjunction with the speaker encoder to obtain the speaker embedding vector, the corresponding process has the following relationship:

[0052] ;

[0053] in, Indicates the The vector corresponding to the speaker of the conversation , Indicates the length is The real number space, Expressing sentences Speaker embedding vector, represents the embedding matrix.

[0054] It should be noted that Only one element is 1, corresponding to the position of the current speaker, and the rest are 0.

[0055] Step 3: Use the multimodal graph convolutional network to perform feature interaction and fusion on the multimodal features to obtain the final comprehensive feature representation.

[0056] In step 3, the multimodal features are interacted and fused using a multimodal graph convolutional network to obtain the final comprehensive feature representation, which specifically includes the following steps:

[0057] Modal nodes are defined for the extracted text token vector representation, the extracted audio features, and the extracted visual representation to obtain text modal nodes, audio modal nodes, and visual modal nodes respectively;

[0058] Fully connect all nodes of the same modality in a multimodal conversation, interconnect nodes of different modalities in different discourses, and construct a graph to obtain an undirected graph.

[0059] Use graph convolutional networks to iteratively update the node features in the constructed undirected graph to obtain the final node representation;

[0060] Perform feature extraction and feature fusion operations on the final node representation in sequence to obtain the fusion features of the discourse;

[0061] The fusion features of the discourse, the speaker embedding vectors and the discourse in the multimodal conversation are concatenated to obtain the final comprehensive feature representation.

[0062] All nodes of the same modality in a multimodal conversation are fully connected, and nodes of different modalities in different discourses are interconnected. Then, a graph is constructed to obtain an undirected graph. The corresponding relationship in the process is as follows:

[0063] ;

[0064] in, represents the constructed undirected graph, Represents the utterance nodes in text modality nodes, audio modality nodes, and visual modality nodes, represents the edge set containing context and modality dependencies;

[0065] In the step of iteratively updating the node features in the constructed undirected graph using the graph convolutional network to obtain the final node representation, the corresponding process has the following relationship:

[0066] ;

[0067] in, Indicates that after The node feature matrix after the layer graph convolutional network, represents the activation function, and Represent two hyperparameters, represents the identity matrix, represents the normalized graph Laplacian matrix of an undirected graph, Indicates the The node feature matrix after the layer graph convolutional network, represents the original feature representation of the node, represents the learnable weight matrix, Represents the layer index of the graph convolutional network, The text modal node representing the first utterance passes through the Feature representation after layer graph convolutional network, The audio modal node representing the first utterance passes through the Feature representation after layer graph convolutional network, The visual modality node representing the first utterance passes through the Feature representation after layer graph convolutional network, Indicates the The text modal node of the discourse passes through the Feature representation after layer graph convolutional network, Indicates the The audio modal node of the speech passes through the Feature representation after layer graph convolutional network, Indicates the The visual modality node of the discourse passes through the Feature representation after layer graph convolutional network;

[0068] In the step of performing feature extraction and feature fusion operations on the final node representation in sequence to obtain the fusion features of the discourse, the corresponding process has the following relationship:

[0069] ;

[0070] in, Indicates the The final feature of the textual modality of an utterance is Indicates the The final features of the audio modality of the utterance, Indicates the The final feature of the visual modality of an utterance is, represents the feature concatenation operation, Indicates the The fusion characteristics of the discourse;

[0071] In the step of concatenating the fusion features of the utterance, the speaker embedding vector, and the utterances in the multimodal conversation to obtain the final comprehensive feature representation, the corresponding process has the following relationship:

[0072] ;

[0073] in, represents the final comprehensive feature representation, Represents a splicing operation.

[0074] Step 4: Use the feedforward network to classify and predict the final comprehensive feature representation, and obtain the emotional representation, emotional prediction results, cause representation, and cause prediction results respectively.

[0075] In step 4, the final comprehensive feature representation is classified and predicted using a feedforward network to obtain the emotional representation, emotional prediction results, cause representation, and cause prediction results, respectively. The specific steps include the following:

[0076] The final comprehensive feature representation is processed using a feedforward network to obtain the emotional representation;

[0077] Predict the emotional expression and obtain the emotional prediction result;

[0078] The final comprehensive feature representation is processed by a feedforward network to obtain the cause representation;

[0079] The cause representation is predicted to obtain the cause prediction result.

[0080] The final comprehensive feature representation is processed using a feedforward network mechanism to obtain the emotional representation. The corresponding relationship is as follows:

[0081] ;

[0082] in, To express emotions, represents a two-layer feedforward network;

[0083] In the step of predicting the emotional expression and obtaining the emotional prediction result, the corresponding process has the following relationship:

[0084] ;

[0085] in, represents the probability distribution of sentiment prediction, It means that after normalization, represents the predicted sentiment label, Indicates that the maximum value in the independent variable is returned. Represents the category index;

[0086] It should be noted that Represents the category index, that is, traversing all possible emotion categories to find the target with the highest probability.

[0087] When the final comprehensive feature representation is processed by the feedforward network, the representation of the cause is obtained. The relationship between the corresponding process is as follows:

[0088] ;

[0089] in, Indication of reasons, represents another two-layer feedforward network;

[0090] In the step of predicting the cause-related representation and obtaining the cause-related prediction result, the corresponding process has the following relationship:

[0091] ;

[0092] in, Represents the predicted reason label probability.

[0093] Step 5: Use the emotion representation and the cause representation to perform candidate emotion-cause pairing, use the speaker relationship matrix to guide the extraction of candidate emotion-cause pairings, and use the expert network and the gated network to perform classification prediction to obtain the prediction results of the emotion-cause pair;

[0094] Construct sentiment prediction loss based on sentiment prediction results;

[0095] Construct cause identification loss based on the prediction results of cause aspects;

[0096] Construct the emotion-reason pair classification loss based on the final emotion-reason pair;

[0097] The prediction model is optimized using sentiment prediction loss, cause identification loss, and sentiment-cause pair classification loss to obtain the optimized prediction model;

[0098] The multimodal conversation is input into the optimized prediction model to obtain the final prediction results.

[0099] In step 5, the emotion representation and the cause representation are used to perform candidate emotion-cause pairings. The speaker relationship matrix is ​​used to guide the extraction of candidate emotion-cause pairs. The expert network and the gated network are used to perform classification prediction to obtain the prediction results of the emotion-cause pairs. The specific steps include the following:

[0100] Constructing candidate pairs of candidate cause utterances in the emotion representation and the cause representation to obtain constructed candidate emotion-cause pairs;

[0101] The representation of the cause and the representation of the emotion are concatenated to obtain the features of the candidate pairings;

[0102] A role relationship matrix is ​​constructed based on the speaker relationship to obtain a constructed role relationship matrix, and a projection operation is performed on the constructed candidate emotion-reason pairs using the constructed role relationship matrix to obtain speaker relationship features;

[0103] The features of the candidate pairings and the speaker relationship features are fused using a gating mechanism to obtain the final fused feature representation;

[0104] The final fusion feature representation is input into the expert network to predict the constructed candidate emotion-cause pairing to obtain the predicted value output by the expert network;

[0105] The final fusion feature representation is input into the gating network to predict the constructed candidate emotion-cause pairing to obtain the predicted value output by the gating network;

[0106] Create one-hot labels based on sentiment-reason pair relationships and combine them with the probability distribution output by the gating network

[0107] Perform re-prediction to obtain the final comprehensive routing probability distribution;

[0108] The final comprehensive routing probability distribution is combined with the predicted value output by the expert network to perform classification prediction and obtain the classification prediction result;

[0109] The classification prediction results are thresholded and the prediction results of the emotion-reason pairs are obtained.

[0110] The candidate pairs of emotion representation and cause representation are constructed to obtain the constructed candidate emotion-cause pairs. The relationship between the corresponding process is as follows:

[0111] ;

[0112] in, Expressing sentences and sentences The candidate pairs are composed of Expressing sentences The speaker, Expressing sentences The speaker, Candidate representation of the cause utterance;

[0113] In the step of concatenating the cause representation and the emotion representation to obtain the features of the candidate pairing, the corresponding process has the following relationship:

[0114] ;

[0115] in, Characteristic of candidate pairings;

[0116] In the steps of constructing a role relationship matrix based on speaker relationships, obtaining a constructed role relationship matrix, and performing a projection operation on the constructed candidate emotion-reason pairs using the constructed role relationship matrix to obtain speaker relationship features, the corresponding process has the following relationship:

[0117] ;

[0118] in, Represents the constructed role relationship matrix, Expressing sentences Speaker embedding vector, and Represent two different learnable weight matrices, Indicates speaker relationship characteristics;

[0119] In the step of fusing the candidate pairing features and speaker relationship features using a gating mechanism to obtain the final fused feature representation, the corresponding process has the following relationship:

[0120] ;

[0121] in, represents the calculated gate value, represents the first learnable parameter, represents the second learnable parameter, represents the final fusion feature representation;

[0122] In the step of inputting the final fusion feature representation into the expert network and predicting the constructed candidate emotion-cause pair to obtain the predicted value output by the expert network, the corresponding process has the following relationship:

[0123] ;

[0124] in, Indicates the Expert network for sentences and sentences Candidate pairs The predicted value of the output, represents the third learnable parameter, represents the fourth learnable parameter, express activation function, represents the fifth learnable parameter, represents the sixth learnable parameter;

[0125] In the step of inputting the final fusion feature representation into the gating network and predicting the constructed candidate emotion-cause pair to obtain the predicted value output by the gating network, the corresponding process has the following relationship:

[0126] ;

[0127] in, represents the distribution on 5 experts, represents the 7th learnable parameter, Indicates the 8th learnable parameter;

[0128] It should be noted that The five experts deal with five different corresponding situations respectively. In the case of actual reasoning, the model cannot determine the labels of the candidate pairs, and the gating network is used to adjust the probability of the experts predicting the corresponding candidate pairs.

[0129] In the step of creating a one-hot label based on the emotion-reason pair relationship and re-predicting it based on the probability distribution output by the gating network to obtain the final comprehensive routing probability distribution, the corresponding process has the following relationship:

[0130] ;

[0131] in, represents the final comprehensive routing probability distribution, represents the equilibrium parameter, Represents the guiding probability distribution based on the pair relationship category;

[0132] In the step of combining the final comprehensive routing probability distribution with the predicted value output by the expert network for classification prediction and obtaining the classification prediction result, the corresponding process has the following relationship:

[0133] ;

[0134] in, Representing emotion-reason pairings The final classification prediction probability, Indicates the The routing probability of an expert, Indicates the The predicted probabilities of emotion-reason pairs by experts.

[0135] Based on the prediction results of sentiment, the sentiment prediction loss is constructed. The relationship between the corresponding process is as follows:

[0136] ;

[0137] in, represents the sentiment prediction loss, represents the logarithmic function, Indicates the The emotional truth label of each utterance;

[0138] In the step of constructing the cause identification loss based on the cause-related prediction results, the corresponding process has the following relationship:

[0139] ;

[0140] in, Indicates cause identification loss, Indicates the The true label of the reason for each utterance;

[0141] In the step of constructing the emotion-reason pair classification loss based on the final emotion-reason pair, the relationship between the corresponding process is as follows:

[0142] ;

[0143] in, represents the sentiment-reason pair classification loss, Representing emotion-reason pairings The true label.

[0144] Furthermore, there is a total loss function, and the corresponding process relationship is as follows:

[0145] ;

[0146] in, represents the total loss function.

[0147] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0148] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0149] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations, characterized by: The method comprises the following steps: Step 1: Build a feature extractor based on the feature extraction mechanism. The feature extractor includes a pre-trained language model, a bidirectional long short-term memory network, an expanded three-dimensional convolutional neural network model, an embedding layer, and a speaker encoder. Build a graph convolutional network based on the feature interaction and fusion mechanism, a gated network based on the gating mechanism, and an expert network based on the expert mixture mechanism. The feature extractor, graph convolutional network, feedforward network, expert network, and gated network constitute the prediction model. Step 2: Input the multimodal conversation into the feature extractor, and extract features of the multimodal conversation in combination with the speaker encoder to obtain multimodal features; Step 3: Use the multimodal graph convolutional network to perform feature interaction and fusion on the multimodal features to obtain the final comprehensive feature representation; Step 4: Use the feedforward network to classify and predict the final comprehensive feature representation, and obtain the emotional representation, emotional prediction results, cause representation, and cause prediction results respectively; Step 5: Use the emotion representation and the cause representation to perform candidate emotion-cause pairing, use the speaker relationship matrix to guide the extraction of candidate emotion-cause pairings, and use the expert network and the gated network to perform classification prediction to obtain the prediction results of the emotion-cause pair; Construct sentiment prediction loss based on sentiment prediction results; Construct cause identification loss based on the prediction results of cause aspects; Construct the emotion-reason pair classification loss based on the final emotion-reason pair; The prediction model is optimized using sentiment prediction loss, cause identification loss, and sentiment-cause pair classification loss to obtain the optimized prediction model; The multimodal dialogue is input into the optimized prediction model to obtain the final prediction result.

2. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 1 is characterized in that: In step 2, the multimodal conversation is input into the feature extractor, and the multimodal conversation is subjected to feature extraction in combination with the speaker encoder to obtain multimodal features, which specifically includes the following steps: Input a multimodal conversation, perform sequence concatenation on the multimodal conversation, and use markers to separate the input sequence. Input the input sequence into the pre-trained language model and perform feature extraction in combination with the speaker encoder to obtain the extracted text token vector representation; Use an open source toolkit to extract features from multimodal conversations to obtain raw audio features. Input the raw audio features into a bidirectional long short-term memory network and combine them with a speaker encoder to extract features and obtain extracted audio features. The dilated 3D convolutional neural network model is used to extract features from multimodal dialogues to obtain extracted raw visual features. The extracted raw visual features are then subjected to feature extraction using a bidirectional long short-term memory network combined with a speaker encoder to obtain an extracted visual representation. The multimodal dialogue is represented by a vector to obtain the represented vector, which is passed through the embedding layer and mapped with the speaker encoder to obtain the speaker embedding vector.

3. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 2 is characterized in that: Input multimodal discourse, the corresponding process has the following relationship: ； in, Represents a multimodal conversation, Indicates the first speaker, Indicates the first utterance, Indicates the A speaker, Indicates the a discourse; In the steps of concatenating multimodal dialogue sequences, separating them using markers to obtain input sequences, feeding the input sequences into a pre-trained language model, and extracting features using a speaker encoder to obtain the extracted text token vector representation, the corresponding process has the following relationship: ； in, represents the entire sequence input to the pre-trained language model, Represents the start token in the pre-trained language model, Represents the end marker in the pre-trained language model, represents the first token in the first text utterance, represents the first token in the second text utterance, Indicates the first start tag The vector representation of Indicates the second start tag The vector representation of Indicates the end tag The vector representation of Indicates that it has been processed by the pre-trained language model. Indicates the start tag Vector representation of ; In the steps of extracting features from multimodal conversations using an open source toolkit to obtain extracted raw audio features, inputting the raw audio features into a bidirectional long short-term memory network, and combining them with a speaker encoder to extract features to obtain extracted audio features, the corresponding process has the following relationship: ； in, Indicates the The original audio features of the utterance, Indicates that the feature extraction is done by the open source toolkit. Indicates the The audio features of an utterance, Indicates that it has been processed by a bidirectional long short-term memory network; In the steps of extracting features from a multimodal conversation using an expanded 3D convolutional neural network model to obtain extracted raw visual features, and extracting features from the extracted raw visual features using a bidirectional long short-term memory network combined with a speaker encoder to obtain an extracted visual representation, the corresponding processes have the following relationship: ； in, Indicates the The original visual features of each utterance, It means that after the expansion of the three-dimensional convolutional neural network model, Indicates the visual features, Indicates feature extraction of original visual features; In the steps of vectorizing the multimodal conversation, obtaining the represented vector, passing the represented vector through the embedding layer, and performing a mapping operation in conjunction with the speaker encoder to obtain the speaker embedding vector, the corresponding process has the following relationship: ； in, Indicates the The vector corresponding to the speaker of the conversation , Indicates the length is The real number space, Expressing sentences Speaker embedding vector, represents the embedding matrix.

4. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 3 is characterized in that: In step 3, the multimodal features are subjected to feature interaction and fusion using a multimodal graph convolutional network to obtain the final comprehensive feature representation, which specifically includes the following steps: Modal nodes are defined for the extracted text token vector representation, the extracted audio features, and the extracted visual representation to obtain text modal nodes, audio modal nodes, and visual modal nodes respectively; Fully connect all nodes of the same modality in a multimodal conversation, interconnect nodes of different modalities in different discourses, and construct a graph to obtain an undirected graph. Use graph convolutional networks to iteratively update the node features in the constructed undirected graph to obtain the final node representation; Perform feature extraction and feature fusion operations on the final node representation in sequence to obtain the fusion features of the discourse; The fusion features of the discourse, the speaker embedding vectors and the discourse in the multimodal conversation are concatenated to obtain the final comprehensive feature representation.

5. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 4 is characterized in that: All nodes of the same modality in a multimodal conversation are fully connected, and nodes of different modalities in different discourses are interconnected. Then, a graph is constructed to obtain an undirected graph. The corresponding relationship in the process is as follows: ； in, represents the constructed undirected graph, Represents the utterance nodes in text modality nodes, audio modality nodes, and visual modality nodes, represents the edge set containing context and modality dependencies; In the step of iteratively updating the node features in the constructed undirected graph using the graph convolutional network to obtain the final node representation, the corresponding process has the following relationship: ； in, Indicates that after The node feature matrix after the layer graph convolutional network, represents the activation function, and Represent two hyperparameters, represents the identity matrix, represents the normalized graph Laplacian matrix of an undirected graph, Indicates the The node feature matrix after the layer graph convolutional network, represents the original feature representation of the node, represents the learnable weight matrix, Represents the layer index of the graph convolutional network, The text modal node representing the first utterance passes through the Feature representation after layer graph convolutional network, The audio modal node representing the first utterance passes through the Feature representation after layer graph convolutional network, The visual modality node representing the first utterance passes through the Feature representation after layer graph convolutional network, Indicates the The text modal node of the discourse passes through the Feature representation after layer graph convolutional network, Indicates the The audio modal node of the speech passes through the Feature representation after layer graph convolutional network, Indicates the The visual modality node of the discourse passes through the Feature representation after layer graph convolutional network; In the step of performing feature extraction and feature fusion operations on the final node representation in sequence to obtain the fusion features of the discourse, the corresponding process has the following relationship: ； in, Indicates the The final feature of the textual modality of an utterance is Indicates the The final features of the audio modality of the utterance, Indicates the The final feature of the visual modality of an utterance is, represents the feature concatenation operation, Indicates the The fusion characteristics of the discourse; In the step of concatenating the fusion features of the utterance, the speaker embedding vector, and the utterances in the multimodal conversation to obtain the final comprehensive feature representation, the corresponding process has the following relationship: ； in, represents the final comprehensive feature representation, Represents a splicing operation.

6. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 5 is characterized in that: In step 4, the final comprehensive feature representation is classified and predicted using a feedforward network to obtain the emotional representation, the emotional prediction result, the cause representation, and the cause prediction result, respectively. Specifically, the following steps are included: The final comprehensive feature representation is processed using a feedforward network to obtain the emotional representation; Predict the emotional expression and obtain the emotional prediction result; The final comprehensive feature representation is processed by a feedforward network to obtain the cause representation; The cause representation is predicted to obtain the cause prediction result.

7. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 6, characterized in that: The final comprehensive feature representation is processed using a feedforward network mechanism to obtain the emotional representation. The corresponding relationship is as follows: ； in, To express emotions, represents a two-layer feedforward network; In the step of predicting the emotional expression and obtaining the emotional prediction result, the corresponding process has the following relationship: ； in, represents the probability distribution of sentiment prediction, It means that after normalization, represents the predicted sentiment label, Indicates that the maximum value in the independent variable is returned. Represents the category index; When the final comprehensive feature representation is processed by the feedforward network, the representation of the cause is obtained. The relationship between the corresponding process is as follows: ； in, Indication of reasons, represents another two-layer feedforward network; In the step of predicting the cause-related representation and obtaining the cause-related prediction result, the corresponding process has the following relationship: ； in, Represents the predicted reason label probability.

8. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 7 is characterized in that: In step 5, candidate emotion-cause pairs are paired using the emotion representation and the cause representation, the speaker relationship matrix is ​​used to guide the extraction of candidate emotion-cause pairs, and the expert network and the gated network are used to perform classification prediction to obtain the prediction results of the emotion-cause pairs. Specifically, the steps include: Constructing candidate pairs of candidate cause utterances in the emotion representation and the cause representation to obtain constructed candidate emotion-cause pairs; The representation of the cause and the representation of the emotion are concatenated to obtain the features of the candidate pairings; A role relationship matrix is ​​constructed based on the speaker relationship to obtain a constructed role relationship matrix, and a projection operation is performed on the constructed candidate emotion-reason pairs using the constructed role relationship matrix to obtain speaker relationship features; The features of the candidate pairings and the speaker relationship features are fused using a gating mechanism to obtain the final fused feature representation; The final fusion feature representation is input into the expert network to predict the constructed candidate emotion-cause pairing to obtain the predicted value output by the expert network; The final fusion feature representation is input into the gating network to predict the constructed candidate emotion-cause pairing to obtain the predicted value output by the gating network; Create one-hot labels based on sentiment-reason pair relationships and combine them with the probability distribution output by the gating network Perform re-prediction to obtain the final comprehensive routing probability distribution; The final comprehensive routing probability distribution is combined with the predicted value output by the expert network to perform classification prediction and obtain the classification prediction result; The classification prediction results are thresholded and the prediction results of the emotion-reason pairs are obtained.

9. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 8, characterized in that: The candidate pairs of emotion representation and cause representation are constructed to obtain the constructed candidate emotion-cause pairs. The relationship between the corresponding process is as follows: ； in, Expressing sentences and sentences The candidate pairs are composed of Expressing sentences The speaker, Expressing sentences The speaker, Candidate representation of the cause utterance; In the step of concatenating the cause representation and the emotion representation to obtain the features of the candidate pairing, the corresponding process has the following relationship: ； in, Characteristic of candidate pairings; In the steps of constructing a role relationship matrix based on speaker relationships, obtaining a constructed role relationship matrix, and performing a projection operation on the constructed candidate emotion-reason pairs using the constructed role relationship matrix to obtain speaker relationship features, the corresponding process has the following relationship: ； in, Represents the constructed role relationship matrix, Expressing sentences Speaker embedding vector, and Represent two different learnable weight matrices, Indicates speaker relationship characteristics; In the step of fusing the candidate pairing features and speaker relationship features using a gating mechanism to obtain the final fused feature representation, the corresponding process has the following relationship: ； in, represents the calculated gate value, represents the first learnable parameter, represents the second learnable parameter, represents the final fusion feature representation; In the step of inputting the final fusion feature representation into the expert network and predicting the constructed candidate emotion-cause pair to obtain the predicted value output by the expert network, the corresponding process has the following relationship: ； in, Indicates the Expert network for sentences and sentences Candidate pairs The predicted value of the output, represents the third learnable parameter, represents the fourth learnable parameter, express activation function, represents the fifth learnable parameter, represents the sixth learnable parameter; In the step of inputting the final fusion feature representation into the gating network and predicting the constructed candidate emotion-cause pair to obtain the predicted value output by the gating network, the corresponding process has the following relationship: ； in, represents the distribution on 5 experts, represents the 7th learnable parameter, Indicates the 8th learnable parameter; In the step of creating a one-hot label based on the emotion-reason pair relationship and re-predicting it based on the probability distribution output by the gating network to obtain the final comprehensive routing probability distribution, the corresponding process has the following relationship: ； in, represents the final comprehensive routing probability distribution, represents the equilibrium parameter, Represents the guiding probability distribution based on the pair relationship category; In the step of combining the final comprehensive routing probability distribution with the predicted value output by the expert network for classification prediction and obtaining the classification prediction result, the corresponding process has the following relationship: ； in, Expressing emotion-reason pairings The final classification prediction probability, Indicates the The routing probability of an expert, Indicates the The predicted probabilities of emotion-reason pairs by experts.

10. The method for extracting emotion-reason pairs from multimodal dialogues based on speaker relations according to claim 9, characterized in that: Based on the prediction results of sentiment, the sentiment prediction loss is constructed. The relationship between the corresponding process is as follows: ； in, represents the sentiment prediction loss, represents the logarithmic function, Indicates the The emotional truth label of each utterance; In the step of constructing the cause identification loss based on the cause-related prediction results, the corresponding process has the following relationship: ； in, Indicates cause identification loss, Indicates the The true label of the reason for each utterance; In the step of constructing the emotion-reason pair classification loss based on the final emotion-reason pair, the relationship between the corresponding process is as follows: ； in, represents the sentiment-reason pair classification loss, Expressing emotion-reason pairings The true label.

Images