Multi-modal dialogue emotion recognition method based on attention mechanism and graph neural network

Abstract

The invention relates to the field of emotion recognition, in particular to a multi-modal dialogue emotion recognition method based on an attention mechanism and a graph neural network. The invention provides a multi-modal dialogue emotion recognition model MTG-ERC based on an attention mechanism and a graph neural network, the model constructs a multi-level Transform fusion module, and information interaction in modals and between modals is efficiently captured by using a multi-head self-attention mechanism and a cross-modal attention mechanism. For the limitation of the attention mechanism in the aspect of modeling short-term dependence, a directed multilateral relation graph fusion module is further constructed, and efficient fusion of multi-modal information is realized by means of a directed graph structure and a multilateral relation. Through joint output of the two modules, the model can comprehensively integrate global and local dialogue emotion features, emotion interaction inside and outside the mode is enhanced, and the overall performance of the model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a multimodal emotion recognition method based on an attention mechanism and a graph neural network. Background Art

[0002] With the rapid development of artificial intelligence (AI), the application of conversational technology has expanded beyond basic functionality to provide users with a higher-quality experience and more personalized services. Therefore, enabling machines to understand the emotions of interlocutors is key to advancing AI development. Conversational emotion recognition has become a research hotspot in both academia and industry. Compared to single-modality techniques, multimodal technology can provide a richer source of emotional information.

[0003] In the field of emotion recognition in multimodal conversations, effectively extracting and fusing emotional information from various modes of conversation, particularly intra-modal and inter-modal emotional information, remains a significant challenge. While current mainstream attention-based methods can effectively capture the global context of a conversation, they often neglect modeling local, short-term dependencies, resulting in limitations in capturing subtle fluctuations in emotional change. This is particularly true when emotions in the short-term context frequently fluctuate, making it difficult for existing models to maintain high recognition accuracy. Furthermore, while graph neural network-based methods can meticulously focus on contextual details surrounding an utterance by modeling local graph structures, they are relatively inadequate at capturing long-range dependencies and global emotional features. This makes emotion recognition models less than ideal when faced with emotional fluctuations spanning long timescales. Therefore, current multimodal emotion recognition faces two core challenges: First, accurately capturing interactions within and between modalities. Single modal or local information often cannot fully reflect emotional changes, so effectively fusing information from different modalities is key to improving emotion recognition. The second challenge is how to effectively integrate global and local emotional features, capturing both the long-range dependencies of emotional changes and short-term fluctuations. Existing models fall short in balancing and integrating these two aspects, necessitating a suitable fusion strategy between global information and local details. Summary of the Invention

[0004] In response to the above problems, the present invention proposes a multimodal conversation emotion recognition model based on attention mechanism and graph neural network (MTG-ERC). A multi-level Transformer fusion module combining multi-head self-attention and cross-modal attention mechanism is designed, aiming to capture the emotional features inside and outside the modality more comprehensively through deep multi-level information interaction. The self-attention mechanism can finely model emotional features within a single modality. At the same time, the cross-modal attention mechanism helps to better integrate emotional information between different modalities, thereby comprehensively improving the accuracy of emotion recognition. In addition, in response to the shortcomings of short-term dependency modeling, a directed multilateral relationship graph fusion module is designed. Under the guidance of the graph structure, the diverse relationships between different nodes (i.e., discourse units) are captured, so that emotion recognition can not only better focus on local details, but also handle multilateral relationships more flexibly.

[0005] To achieve the above objectives, the present invention adopts a technical solution: a multimodal conversation emotion recognition method based on attention mechanism and graph neural network, comprising the following steps:

[0006] 1) Select a public dataset D and divide it into training set D train and the test set D test ; The data set D is Each sample d in the dataset D i Corresponding to a conversation segment, it consists of a triple composition, Represent the text data, audio data and visual data of the i-th dialogue segment respectively;

[0007] 2) Construct a multimodal conversation emotion recognition model MTG-ERC based on attention mechanism and graph neural network. The MTG-ERC model includes a unimodal feature extraction module, a speaker embedding module, an attention mechanism-based module, a graph neural network-based module and a multimodal emotion classification module. The unimodal feature extraction module consists of a Transformer model and a fully connected network model. The Transformer model is used to extract features from text data, and the fully connected network model is used to extract features from audio data and visual data. The speaker embedding module maps the speaker in the conversation into a vector to enhance the discourse features. The attention mechanism-based module designs intra-modal and inter-modal Transformer networks to capture the interactions within the modality and between different modalities of the multimodal discourse sequence. The graph neural network-based module constructs a directed graph structure of multilateral relationships to improve the model's ability to parse emotions. The multimodal emotion classification module is used to predict emotion categories and output emotion labels for each sentence.

[0008] 3) Using training set D train Train the multimodal conversation emotion recognition model MTG-ERC and calculate the loss function value. The calculation formula is as follows:

[0009]

[0010] Among them, y ik ∈{0,1} represents the true label of the i-th sentence in the k-th sentiment category; Represents the model's predicted probability for the category; N represents the total number of samples in the training set; M represents the total number of categories of sentiment classification.

[0011] 4) Use test set D test Test the performance of the trained MTG-ERC model. When the loss function value stops decreasing on the test set for x consecutive times, stop training, output the trained optimal model weight file best.pt, and go to step 5). Otherwise, go to step 3) and retrain.

[0012] 5) Input the collected multimodal data, use the optimal model weight file best.pt to perform the multimodal emotion recognition task, and output the emotion analysis results.

[0013] Furthermore, the single-modal feature extraction module in the above step 2) includes:

[0014] Contextual representation of text modality The extraction formula is: Where: Represents the i-th text information, is the Transformer parameter to be learned;

[0015] Contextual representation of audio modality The extraction formula is: Where: represents i audio signals, Represents the parameters of the pre-trained fully connected network model;

[0016] Contextual representation of visual modality The extraction formula is: Where: represents i visual signals, Represents the parameters of the pre-trained fully connected network model.

[0017] Furthermore, the speaker embedding module in the above step 2) includes:

[0018] The speaker embedding module is used to enhance the speaker information and map the speaker in the conversation into a vector S emb , the extracted speech features can be enhanced by adding the corresponding speaker embedding, the specific formula is as follows:

[0019] S emb =Embedding(S,M p )

[0020] X τ =ηS emb +x τ ,τ∈{l,a,v}

[0021] Where: S represents the set of speakers; M p represents the total number of participants in the conversation; x τ Respectively represent the global context representation of the entire conversation after extracting text, audio, and visual modal features; X τ represents the global context representation of each modality after speaker embedding enhancement; η∈[0,1] represents the contribution rate.

[0022] Furthermore, the attention mechanism-based module in step 2) above designs intra-modal and inter-modal Transformer networks to simulate the interaction within the multimodal speech sequence and between different modalities, named Intra-Transformer and Inter-Transformer respectively. The Intra-Transformer network represents the global context of each modality as X τ , as Q, K, V input, captures the interaction between the modes of the discourse sequence and enhances the emotional representation of the discourse; Inter-Transformer network sets Q = X τ , K=V=X υ, so that the τ mode obtains the information of the υ mode, thereby capturing the interaction between each mode. The calculation formula is as follows:

[0023] X τ→τ =Intra-Transformer(X τ ,X τ ,X τ )

[0024] X υ→τ =Inter-Transformer(X τ ,X υ ,X υ )

[0025] Where: τ∈{l,a,v},υ∈{l,a,v}-{τ}.

[0026] Furthermore, the module based on graph neural network in step 2) above includes the following:

[0027] 5.1) Each dialogue segment u i Generate three modal nodes, respectively recorded as The feature vector representation of the node is

[0028] 5.2) Express the edge set as Represents Two points and relationship type r ij ∈R interaction, where R intra It represents the internal connection between the three modes in the same sentence, reflecting the interaction of multiple modes; each mode also has a self-connected link to strengthen the information of the mode itself. The following formula represents the nine R intra Type of edge relationship;

[0029]

[0030] R inter It represents the context between the same modal discourse, including the influence of past and future discourse on the current discourse, and reflects the interaction between different time nodes, as shown below:

[0031]

[0032] Where: → and ← represent the connection relationship with the past and the future respectively; P and F represent the number of sentences connected with the past and the future respectively;

[0033] 5.3) Use RGCN network to distinguish different types of edges and nodes: For each relationship type, the node representation is represented by a mapping function f(H,W r ) to infer, where Wr The weighted matrix aggregates all 15 edge types; specifically, the representation formula for the i-th sentence is:

[0034]

[0035] Where: N r (i) represents all neighbor nodes that have relationship r with node i; W r and W0 are learnable parameters in the RGCN network;

[0036] 5.4) Using the Graph Transformer network to further extract graph-structured data. In the Graph Transformer network, each layer includes a self-attention mechanism. This self-attention mechanism enables each node to pay attention to the information of other nodes in the graph, thereby enhancing the ability to express features.

[0037] The data obtained from the RGCN network Transformed by the Graph Transformer network:

[0038]

[0039] Where: N r (i) is the set of nodes adjacent to node i; W1 and W2 are learnable parameters in the Graph Transformer network; C is the number of connected multi-head attention heads.

[0040] Attention coefficient of node j It is calculated by the softmax activation function:

[0041]

[0042] Where W3 and W4 are learnable parameters; d represents the dimension of the feature vector, which is used to scale the attention score to prevent excessive values from affecting training stability.

[0043] After aggregating the entire graph, we get a new vector representation:

[0044] Furthermore, the multimodal emotion classification module in step 2) above includes the following:

[0045] 6.1) The global and local context representations generated by the multi-level Transformer fusion module and the directed multi-edge graph fusion module are concatenated together as shown in the following formula:

[0046] H=Fusion([X,G])

[0047] Where: Fusion represents the connection method, X represents the output features of the multi-level Transformer fusion module, and G represents the output features of the directed multilateral graph fusion module.

[0048] 6.2) Input H into a fully connected layer, first perform linear transformation and weighted summation, and then process it through the ReLU activation function, as shown in the following formula:

[0049] p i =ReLU(φ0h i +b0)

[0050] Where: h i represents the fusion feature of the i-th sample; φ0 is the weight matrix of the fully connected layer; b0 is the bias term; ReLU(·) is the activation matrix of the rectified linear unit.

[0051] 6.3) The softmax function is used to ensure that the predicted value of each emotion category is in the range of 0 to 1, and the sum of the probabilities of all categories is 1, as shown in the following formula:

[0052] q i =softmax(φ1p i +b1)

[0053] Where: φ1 is the weight matrix of the softmax layer; b1 is the bias term, q i is the probability distribution of the i-th sample belonging to the emotion category.

[0054] 6.4) Use the argmax operation to select the category with the highest probability as the final sentiment prediction result. The calculation formula is as follows:

[0055] result=argmax(q i )

[0056] Where: argmax(·) represents the category index corresponding to the maximum probability.

[0057] The present invention provides a method for multimodal conversation emotion recognition and proposes a multimodal conversation emotion recognition model MTG-ERC based on an attention mechanism and a graph neural network. Through a multi-level Transformer fusion module and a directed multilateral graph fusion module, it effectively captures the conversation emotion information within and between modalities, as well as the emotion information of the global and local contexts, effectively improving the accuracy of multimodal conversation emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the multimodal conversation emotion recognition method based on attention mechanism and graph neural network in the present invention.

[0059] Figure 2 This is the overall architecture diagram of the present invention.

[0060] Figure 3 This is the Intra-Transformer structure diagram.

[0061] Figure 4 This is the Inter-Transformer structure diagram. DETAILED DESCRIPTION

[0062] The present invention is further described below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the technical solution and design principle of the present invention are described in detail below only with an optimized technical solution, but the protection scope of the present invention is not limited thereto.

[0063] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, the present invention is a multimodal conversation emotion recognition method based on attention mechanism and graph neural network, which includes the following steps:

[0065] 1) Select a public dataset D and divide it into training set D train and the test set D test ; The data set D is Each sample d in the dataset D i Corresponding to a conversation segment, it consists of a triple composition, represent the text data, audio data, and visual data of the i-th conversation segment respectively.

[0066] In a specific embodiment of the present invention, the public dataset IEMOCAP is selected as the dataset of the multimodal dialogue emotion recognition method of the present invention, and the dataset IEMOCAP is divided into the training set D and the speech set D according to a ratio of 9:1. train and the test set D test .

[0067] 2) If Figure 2As shown, a multimodal conversation emotion recognition model MTG-ERC based on attention mechanism and graph neural network is constructed. The MTG-ERC model includes a unimodal feature extraction module, a speaker embedding module, a module based on attention mechanism, a module based on graph neural network and a multimodal emotion classification module; wherein the unimodal feature extraction module is composed of a Transformer model and a fully connected network model. The Transformer model is used to extract features from text data, and the fully connected network model is used to extract features from audio data and visual data; the speaker embedding module maps the speaker in the conversation into a vector to enhance the discourse features; the module based on attention mechanism designs intra-modal and inter-modal Transformer networks to capture the interactions within the modality and between different modalities of the multimodal discourse sequence; the module based on graph neural network constructs a directed graph structure of multilateral relationships to improve the model's ability to parse emotions; the multimodal emotion classification module is used to predict emotion categories and output emotion labels for each sentence;

[0068] 3) Using training set D train Train the multimodal conversation emotion recognition model MTG-ERC and calculate the loss function value. The calculation formula is as follows:

[0069]

[0070] Among them, y ik ∈{0,1} represents the true label of the i-th sentence in the k-th sentiment category; Represents the model's predicted probability for the category; N represents the total number of samples in the training set; M represents the total number of categories of sentiment classification.

[0071] 4) Use test set D test Test the performance of the trained MTG-ERC model. When the loss function value stops decreasing on the test set for x consecutive times, stop training, output the trained optimal model weight file best.pt, and go to step 5). Otherwise, go to step 3) and retrain.

[0072] 5) Input the collected multimodal data, use the optimal model weight file best.pt to perform the multimodal emotion recognition task, and output the emotion analysis results.

[0073] As a preferred embodiment of the present invention, the single-modal feature extraction module in step 2) includes:

[0074] Contextual representation of text modality The extraction formula is: Where: Represents the i-th text information, is the Transformer parameter to be learned;

[0075] Contextual representation of audio modality The extraction formula is: Where: represents i audio signals, Represents the parameters of the pre-trained fully connected network model;

[0076] Contextual representation of visual modality The extraction formula is: Where: represents i visual signals, Represents the parameters of the pre-trained fully connected network model.

[0077] As a preferred embodiment of the present invention, the speaker embedding module in step 2) includes:

[0078] The speaker embedding module is used to enhance the speaker information and map the speaker in the conversation into a vector S emb , the extracted speech features can be enhanced by adding the corresponding speaker embedding, the specific formula is as follows:

[0079] S emb =Embedding(S,M p )

[0080] X τ =ηS emb +x τ ,τ∈{l,a,v}

[0081] Where: S represents the set of speakers; M p represents the total number of participants in the conversation; x τ Respectively represent the global context representation of the entire conversation after extracting text, audio, and visual modal features; X τ represents the global context representation of each modality after speaker embedding enhancement; η∈[0,1] represents the contribution rate.

[0082] As a preferred embodiment of the present invention, the module based on the attention mechanism in step 2) designs the intra-modal and inter-modal Transformer networks to simulate the interaction within the modality and between different modalities of the multimodal speech sequence, which are named Intra-Transformer and Inter-Transformer respectively, with the following structure: Figure 3 and Figure 4 As shown. The Intra-Transformer network represents the global context of each modality as X τ , as Q, K, V input, captures the interaction between the modes of the discourse sequence and enhances the emotional representation of the discourse; Inter-Transformer network sets Q = Xτ , K=V=X υ , so that the τ mode obtains the information of the υ mode, thereby capturing the interaction between each mode. The calculation formula is as follows:

[0083] X τ→τ =Intra-Transformer(X τ ,X τ ,X τ )

[0084] X υ→τ =Inter-Transformer(X τ ,X υ ,X υ )

[0085] Where: τ∈{l,a,v},υ∈{l,a,v}-{τ}.

[0086] As a preferred embodiment of the present invention, the module based on the graph neural network in step 2) includes the following contents:

[0087] 5.1) Each dialogue segment u i Generate three modal nodes, respectively recorded as The feature vector representation of the node is

[0088] 5.2) Express the edge set as Represents Two points and relationship type r ij ∈R interaction, where R intra It represents the internal connection between the three modes in the same sentence, reflecting the interaction of multiple modes; each mode also has a self-connected link to strengthen the information of the mode itself. The following formula represents the nine R intra Type of edge relationship;

[0089]

[0090] R inter It represents the context between the same modal discourse, including the influence of past and future discourse on the current discourse, and reflects the interaction between different time nodes, as shown below:

[0091]

[0092] Where: → and ← represent the connection relationship with the past and the future respectively; P and F represent the number of sentences connected with the past and the future respectively;

[0093] 5.3) Use RGCN network to distinguish different types of edges and nodes: For each relationship type, the node representation is represented by a mapping function f(H,W r ) to infer, where W r The weighted matrix aggregates all 15 edge types; specifically, the representation formula for the i-th sentence is:

[0094]

[0095] Where: N r (i) represents all neighbor nodes that have relationship r with node i; W r and W0 are learnable parameters in the RGCN network;

[0096] 5.4) Using the Graph Transformer network to further extract graph-structured data. In the Graph Transformer network, each layer includes a self-attention mechanism. This self-attention mechanism enables each node to pay attention to the information of other nodes in the graph, thereby enhancing the ability to express features.

[0097] The data obtained from the RGCN network Transformed by the Graph Transformer network:

[0098]

[0099] Where: N r (i) is the set of nodes adjacent to node i; W1 and W2 are learnable parameters in the Graph Transformer network; C is the number of connected multi-head attention heads.

[0100] Attention coefficient of node j It is calculated by the softmax activation function:

[0101]

[0102] Where W3 and W4 are learnable parameters; d represents the dimension of the feature vector, which is used to scale the attention score to prevent excessive values from affecting training stability.

[0103] After aggregating the entire graph, we get a new vector representation:

[0104] As a preferred embodiment of the present invention, the multimodal emotion classification module in step 2) includes the following contents:

[0105] 6.1) The global and local context representations generated by the multi-level Transformer fusion module and the directed multi-edge graph fusion module are concatenated together as shown in the following formula:

[0106] H=Fusion([X,G])

[0107] Where: Fusion represents the connection method, X represents the output features of the multi-level Transformer fusion module, and G represents the output features of the directed multilateral graph fusion module.

[0108] 6.2) Input H into a fully connected layer, first perform linear transformation and weighted summation, and then process it through the ReLU activation function, as shown in the following formula:

[0109] p i =ReLU(φ0h i +b0)

[0110] Where: h i represents the fusion feature of the i-th sample; φ0 is the weight matrix of the fully connected layer; b0 is the bias term; ReLU(·) is the activation matrix of the rectified linear unit.

[0111] 6.3) The softmax function is used to ensure that the predicted value of each emotion category is in the range of 0 to 1, and the sum of the probabilities of all categories is 1, as shown in the following formula:

[0112] q i =softmax(φ1p i +b1)

[0113] Where: φ1 is the weight matrix of the softmax layer; b1 is the bias term, q i is the probability distribution of the i-th sample belonging to the emotion category.

[0114] 6.4) Use the argmax operation to select the category with the highest probability as the final sentiment prediction result. The calculation formula is as follows:

[0115] result=argmax(q i )

[0116] Where: argmax(·) represents the category index corresponding to the maximum probability.

Claims

1. A multimodal conversation emotion recognition method based on attention mechanism and graph neural network, characterized by: The following steps are involved: 1) Select a public dataset D and divide it into training set D train and the test set D test ; The data set D is , each sample d in the dataset D i Corresponding to a conversation segment, it consists of a triple composition, Represent the text data, audio data and visual data of the i-th dialogue segment respectively; 2) Construct a multimodal conversation emotion recognition model MTG-ERC based on attention mechanism and graph neural network. The MTG-ERC model includes a unimodal feature extraction module, a speaker embedding module, an attention mechanism-based module, a graph neural network-based module and a multimodal emotion classification module. The unimodal feature extraction module consists of a Transformer model and a fully connected network model. The Transformer model is used to extract features from text data, and the fully connected network model is used to extract features from audio data and visual data. The speaker embedding module maps the speaker in the conversation into a vector to enhance the discourse features. The attention mechanism-based module designs intra-modal and inter-modal Transformer networks to capture the interactions within the modality and between different modalities of the multimodal discourse sequence. The graph neural network-based module constructs a directed graph structure of multilateral relationships to improve the model's ability to parse emotions. The multimodal emotion classification module is used to predict emotion categories and output emotion labels for each sentence. 3) Using training set D train Train the multimodal conversation emotion recognition model MTG-ERC and calculate the loss function value. The calculation formula is as follows: Among them, y ik ∈{0,1} represents the true label of the i-th sentence in the k-th sentiment category; Represents the model's predicted probability for the category; N represents the total number of samples in the training set; M represents the total number of categories of sentiment classification; 4) Use test set D test Test the performance of the trained MTG-ERC model. When the loss function value stops decreasing on the test set for x consecutive times, stop training, output the trained optimal model weight file best.pt, and go to step 5). Otherwise, go to step 3) and retrain. 5) Input the collected multimodal data, use the optimal model weight file best.pt to perform the multimodal emotion recognition task, and output the emotion analysis results.

2. The multimodal conversation emotion recognition method based on attention mechanism and graph neural network according to claim 1 is characterized in that: The single-modal feature extraction module in step 2) includes: Contextual representation of text modality The extraction formula is: Where: Represents the i-th text information, is the Transformer parameter to be learned; Contextual representation of audio modality The extraction formula is: Where: represents i audio signals, Represents the parameters of the pre-trained fully connected network model; Contextual representation of visual modality The extraction formula is: Where: represents i visual signals, Represents the parameters of the pre-trained fully connected network model.

3. The multimodal conversation emotion recognition method based on attention mechanism and graph neural network according to claim 1 is characterized in that: The speaker embedding module in step 2) includes: The speaker embedding module is used to enhance the speaker information and map the speaker in the conversation into a vector S emb , the extracted speech features can be enhanced by adding the corresponding speaker embedding, the specific formula is as follows: S emb =Embedding(S,M p ) X τ =ηS emb +x τ ,τ∈{l,a,v} Where: S represents the set of speakers; M p represents the total number of participants in the conversation; x τ Respectively represent the global context representation of the entire conversation after extracting text, audio, and visual modal features; X τ represents the global context representation of each modality after speaker embedding enhancement; η∈[0,1] represents the contribution rate.

4. The multimodal conversation emotion recognition method based on attention mechanism and graph neural network according to claim 1 is characterized in that In step 2), the module based on the attention mechanism designs the intra-modal and inter-modal Transformer networks to simulate the interaction within the multimodal speech sequence and between different modalities, respectively named Intra-Transformer and Inter-Transformer. The Intra-Transformer network represents the global context of each modality as X. τ , as Q, K, V input, captures the interaction between the modes of the discourse sequence and enhances the emotional representation of the discourse; Inter-Transformer network sets Q = X τ , K=V=X υ , so that the τ mode obtains the information of the υ mode, thereby capturing the interaction between each mode. The calculation formula is as follows: X τ→τ =Intra-Transformer(X τ ,X τ ,X τ ) X υ→τ =Inter-Transformer(X τ ,X υ ,X υ ) Where: τ∈{l,a,v},υ∈{l,a,v}-{τ}.

5. The multimodal conversation emotion recognition method based on attention mechanism and graph neural network according to claim 1 is characterized in that: The module based on graph neural network in step 2) includes the following contents: 5.1) Each dialogue segment u i Generate three modal nodes, respectively recorded as The feature vector representation of the node is 5.2) Express the edge set as Represents Two points and relationship type r ij ∈R interaction, where R intra It represents the internal connection between the three modes in the same sentence, reflecting the interaction of multiple modes; each mode also has a self-connected link to strengthen the information of the mode itself. The following formula represents the nine R intra Type of edge relationship; R inter It represents the context between the same modal discourse, including the influence of past and future discourse on the current discourse, and reflects the interaction between different time nodes, as shown below: Where: → and ← represent the connection relationship with the past and the future respectively; P and F represent the number of sentences connected with the past and the future respectively; 5.3) Use RGCN network to distinguish different types of edges and nodes: For each relationship type, the node representation is represented by a mapping function f(H,W r ) to infer, where W r The weighted matrix aggregates all 15 edge types; specifically, the representation formula for the i-th sentence is: Where: N r (i) represents all neighbor nodes that have relationship r with node i; W r and W0 are learnable parameters in the RGCN network; 5.4) Using the Graph Transformer network to further extract graph-structured data. In the Graph Transformer network, each layer includes a self-attention mechanism. This self-attention mechanism enables each node to pay attention to the information of other nodes in the graph, thereby enhancing the ability to express features. The data obtained from the RGCN network Transformed by the Graph Transformer network: Where: N r (i) is the set of nodes adjacent to node i; W1 and W2 are learnable parameters in the Graph Transformer network; C is the number of connected multi-head attention heads; Attention coefficient of node j It is calculated by the softmax activation function: Where: W3 and W4 are learnable parameters; d represents the dimension of the feature vector, which is used to scale the attention score to prevent excessive values from affecting training stability; After aggregating the entire graph, we get a new vector representation:

6. The multimodal conversation emotion recognition method based on attention mechanism and graph neural network according to claim 1 is characterized in that The multimodal emotion classification module in step 2) includes the following contents: 6.1) The global and local context representations generated by the multi-level Transformer fusion module and the directed multi-edge graph fusion module are concatenated together as shown in the following formula: H=Fusion([X,G]) Where: Fusion represents the connection method, X represents the output features of the multi-level Transformer fusion module, and G represents the output features of the directed multilateral graph fusion module; 6.2) Input H into a fully connected layer, first perform linear transformation and weighted summation, and then process it through the ReLU activation function, as shown in the following formula: p i =ReLU(φ0h i +b0) Where: h i represents the fusion feature of the i-th sample; φ0 is the weight matrix of the fully connected layer; b0 is the bias term; ReLU(·) is the activation matrix of the rectified linear unit; 6.3) The softmax function is used to ensure that the predicted value of each emotion category is in the range of 0 to 1, and the sum of the probabilities of all categories is 1, as shown in the following formula: q i =softmax(φ1p i +b1) Where: φ1 is the weight matrix of the softmax layer; b1 is the bias term, q i is the probability distribution of the i-th sample belonging to the emotion category; 6.4) Use the argmax operation to select the category with the highest probability as the final sentiment prediction result. The calculation formula is as follows: result=argmax(q i ) Where: argmax(·) represents the category index corresponding to the maximum probability.

Images