A multimodal conversation emotion recognition method based on cross-modal feature complementation and directed graph learning

Abstract

The present invention discloses a multimodal conversation emotion recognition method based on cross-modal feature complementarity and directed graph learning, and relates to the technical field of conversation emotion recognition. The method comprises: acquiring video conversation data, performing feature extraction on the video conversation data to obtain a first feature; encoding the first feature to obtain a second feature; constructing a cross-modal heterogeneous graph based on the second feature, and obtaining an attention weight based on the cross-modal heterogeneous graph; updating the cross-modal heterogeneous graph based on the attention weight to obtain a final cross-modal heterogeneous graph, extracting complementary feature information of the final cross-modal heterogeneous graph, and fusing features of the final cross-modal heterogeneous graph based on the complementary feature information to obtain enhanced features; constructing a multimodal directed graph based on the enhanced features; obtaining emotion features based on the multimodal directed graph, transmitting the emotion features to a pre-trained emotion classifier for prediction, and obtaining a predicted emotion result, which can solve the problems of conflict and heterogeneity in multimodal data fusion in the existing ERC method, as well as ignoring the interaction between the modes, resulting in loss of modal information and insufficient modal fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of conversation emotion recognition, and in particular to a multimodal conversation emotion recognition method based on cross-modal feature complementarity and directed graph learning. Background Art

[0002] In human-computer interaction, machines can perceive and recognize human emotions, which can promote emotional communication, enhance machines' understanding of users' emotional needs, and promote more intelligent and personalized emotional interactions. Emotion Recognition in Conversation (ERC) is an important research direction in the field of emotion recognition, which is dedicated to identifying and understanding emotions expressed in verbal communication. ERC emphasizes that the conversation context is a complex interaction of various modalities, including spoken language, facial expressions, body language, and text information that may be exchanged in the conversation. With the rapid development of artificial intelligence, the application of ERC is expanding to areas such as dialogue generation, recommendation systems, opinion mining, and medical diagnosis.

[0003] In daily conversations, emotions often flow between contexts, which indicates that current emotions are affected by context. Therefore, many existing ERC methods focus on modeling conversation context information and capturing long-range conversation context dependencies. For example, ConGCN (Compositional Graph Convolutional Network) models the dependency between context information and speaker information by constructing a heterogeneous graph; MMGCN (Multimodal Graph Convolutional Network) integrates the discourse nodes of three modalities into a GNN (Graph Neural Network) to model multimodal information and long-range context dependency information, but it will bring additional noise; COGMEN (Multimodal Emotion Recognition Architecture Based on Contextualized Graph Neural Network) establishes a directed graph structure for the dependency between the speaker self and the speaker, and then uses a graph convolutional network to capture the dependency.

[0004] However, due to the differences in feature variations and data distribution between different modalities, different modalities have unique characteristics. Fusion of different modalities in the graph may result in conflicts in multimodal information and be plagued by data heterogeneity problems. In addition, the above method directly connects the features of each modality for fusion in a simple form, ignoring the interaction between the modalities, resulting in loss of modal information and insufficient modal fusion. Summary of the invention

[0005] In order to solve the conflicts and heterogeneity in multimodal data fusion in the existing ERC method, and to solve the problems of ignoring the interaction between each modality, resulting in loss of modal information and insufficient modal fusion, this embodiment provides a multimodal conversation emotion recognition method based on cross-modal feature complementarity and directed graph learning, the method comprising: acquiring video conversation data, performing feature extraction on the video conversation data to obtain a first feature; encoding the first feature to obtain a second feature; constructing a cross-modal heterogeneous graph based on the second feature, and obtaining an attention weight based on the cross-modal heterogeneous graph; updating the cross-modal heterogeneous graph based on the attention weight to obtain a final cross-modal heterogeneous graph, extracting complementary feature information of the final cross-modal heterogeneous graph, and based on the complementary feature information, fusing the features of the final cross-modal heterogeneous graph to obtain enhanced features; constructing a multimodal directed graph based on the enhanced features; obtaining emotional features based on the multimodal directed graph, and transmitting the emotional features to a pre-trained emotion classifier for prediction to obtain a predicted emotional result.

[0006] The principle of the invention is as follows: by constructing a cross-modal heterogeneous graph containing two modalities for each encoded modality, learning the interactive information between different modalities through a collaborative attention mechanism, so that each modality contains complementary information of the other two modalities, and fusing the features related to the same modality to obtain more comprehensive complementary information, thereby obtaining enhanced feature representations of each modality, and effectively alleviating the conflict and heterogeneity problems in multimodal data fusion; constructing a multimodal directed graph based on speaker relationships based on each enhanced modality, performing different processing on the interactions between nodes occurring in different time sequences in the graph, using a sliding window to control the past nodes and future nodes of the current node, reducing the loss of modal information, making the nodes more closely related, and making the modal fusion more complete; capturing the contextual dependencies between and within speakers in the directed graph, effectively capturing the cross-modal interactions and time dependencies at the discourse level in the dialogue context, and improving the accuracy of dialogue emotion recognition.

[0007] Furthermore, the first feature includes a first text modality feature, a first video modality feature and a first audio modality feature, and the specific steps of obtaining the first feature include: performing feature extraction on an audio sequence in the video conversation data to obtain the first audio modality feature; performing feature extraction on facial expression changes and gesture changes in the video conversation data to obtain the first video modality feature; performing feature extraction on a word vector of each word in the video conversation data to obtain the first text modality feature; and obtaining the first feature based on the first text modality feature, the first video modality feature and the first audio modality feature.

[0008] Furthermore, the second feature includes a second text modality feature, a second video modality feature and a second audio modality feature, and the specific steps for obtaining the second feature include: encoding the first video modality feature and the first audio modality feature based on a normalization layer and a fully connected layer to obtain the second video modality feature and the second audio modality feature, respectively; encoding the first text modality feature based on a normalization layer and a bidirectional LSTM layer to obtain the second text modality feature; obtaining the second feature based on the second text modality feature, the second video modality feature and the second audio modality feature.

[0009] Further, the cross-modal heterogeneous graph includes a first cross-modal heterogeneous graph, a second cross-modal heterogeneous graph and a third cross-modal heterogeneous graph, and the specific steps of constructing the cross-modal heterogeneous graph include: based on the second feature, obtaining a first node of a text modality, a second node of a video modality and a third node of an audio modality; obtaining a first node set and a first connection edge set based on the first node and the second node; constructing the first cross-modal heterogeneous graph based on the first node set and the first connection edge set; obtaining a second node set and a second connection edge set based on the first node and the third node; constructing the second cross-modal heterogeneous graph based on the second node set and the second connection edge set; obtaining a third node set and a third connection edge set based on the second node and the third node; constructing the third cross-modal heterogeneous graph based on the third node set and the third connection edge set; obtaining the cross-modal heterogeneous graph based on the first cross-modal heterogeneous graph, the second cross-modal heterogeneous graph and the third cross-modal heterogeneous graph.

[0010] Furthermore, the specific steps of obtaining the final cross-modal heterogeneous graph include: obtaining neighbor nodes of the fourth node in the cross-modal heterogeneous graph, aggregating the neighbor nodes based on the attention weights to obtain an aggregation result, updating the fourth node based on the aggregation result to obtain a fifth node; and updating the cross-modal heterogeneous graph based on the fifth node to obtain the final cross-modal heterogeneous graph. The aggregated features of the neighbor nodes are combined with the features of the own nodes to update their feature representation, so that the modal fusion is more complete and the feature representation is richer.

[0011] Furthermore, the complementary feature information includes: a first complementary feature for converting the video modality into the text modality, and a second complementary feature for converting the text modality into the video modality; a third complementary feature for converting the audio modality into the text modality, and a fourth complementary feature for converting the text modality into the audio modality; a fifth complementary feature for converting the video modality into the audio modality, and a sixth complementary feature for converting the audio modality into the video modality.

[0012] Furthermore, the specific step of obtaining the enhanced feature includes: based on the complementary feature information and the final cross-modal heterogeneous graph, the features of the nodes involving the same modality are fused to obtain the enhanced feature. Each node in the graph can receive supplementary information of an additional modality, and the remaining modalities are further fused into the current feature. After completing the interaction between the two modalities, the features involving the same modality are fused to obtain more comprehensive complementary information.

[0013] Furthermore, the specific steps of constructing a multimodal directed graph include: obtaining a discourse node based on the enhanced features, obtaining a discourse edge and a relationship type of the discourse edge based on the discourse node, the discourse node includes a text discourse node, a video discourse node and an audio discourse node, the discourse edge includes an external edge between discourse nodes of different modalities and an internal edge between discourse nodes of the same modality, and the relationship type of the edge includes an external edge type and an internal edge type; based on the time sequence of the discourse node, controlling the M past discourse nodes and N future discourse nodes of the discourse node to obtain the multimodal directed graph, where M and N both represent integers greater than or equal to 1. Construct a directed graph structure based on speaker relations to capture the time sequence relationship between discourses in a conversation and the interactive relationship between different speakers; perform different processing on the interactions between nodes occurring in different time sequences in the graph, use a sliding window to control the past nodes and future nodes of the current node, reduce the loss of modal information, make the nodes more closely related, and make the modal fusion more complete.

[0014] Furthermore, the specific steps of obtaining the sentiment feature include: extracting the dependency between internal nodes and adjacent nodes in the discourse node to obtain the third feature; based on the third feature, extracting local information and global information of the multimodal directed graph to obtain the fourth feature; performing neighbor sampling and aggregation on the fourth feature to obtain the sentiment feature. By utilizing the interaction and changes between discourse and modality in the multimodal graph, rich representations are extracted from node features to improve the representation of nodes.

[0015] A dependency is a relationship where a change in one node will affect another node.

[0016] Furthermore, the first calculation formula for obtaining the third feature is:

[0017]

[0018] in, represents the third characteristic, ω top and ω t Both represent learnable parameters, Represents the characteristics of the discourse node, P t(i) Indicates that the node under the relation t∈R The set of neighbor indices of |P t(i) | represents the normalization constant, Representation Node The features of the neighbor nodes of , η represents three different modalities, including text modality, video modality and audio modality, and i and m both represent serial numbers.

[0019] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0020] 1. By constructing a cross-modal heterogeneous graph containing two modalities for each encoded modality, the interactive information between different modalities is learned through the collaborative attention mechanism, so that each modality contains the complementary information of the other two modalities, and the features related to the same modality are fused to obtain more comprehensive complementary information, thereby obtaining enhanced feature representations of each modality, effectively alleviating the conflict and heterogeneity problems in multimodal data fusion.

[0021] 2. Based on the enhanced modalities, a multimodal directed graph based on speaker relations is constructed. Different processes are performed on the interactions between nodes that occur in different time sequences in the graph. A sliding window is used to control the past nodes and future nodes of the current node to reduce the loss of modal information, make the nodes more closely related, and make the modal fusion more complete.

[0022] 3. Capture the contextual dependencies between and within speakers in a multimodal directed graph, effectively capture the cross-modal interactions and temporal dependencies at the discourse level in the dialogue context, and improve the accuracy of dialogue emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0024] Figure 1 It is a flowchart of a multimodal dialogue emotion recognition method based on cross-modal feature complementarity and directed graph learning in the present invention. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0027] Embodiment 1

[0028] refer to Figure 1 This embodiment provides a multimodal conversation emotion recognition method based on cross-modal feature complementation and directed graph learning, the method comprising:

[0029] S1. Obtain video conversation data, perform feature extraction on the video conversation data, and obtain a first feature, where the first feature includes a first text modality feature, a first video modality feature, and a first audio modality feature. The specific steps include:

[0030] Using the openSMILE tool to extract features from the audio sequence in the video conversation data to obtain the first audio modality features;

[0031] Using a pre-trained 3D-CNN (three-dimensional convolutional neural network) model to extract features of facial expression changes and gesture changes in the video conversation data to obtain the first video modality features;

[0032] Using a pre-trained RoBERTa model (RoBERTa model is an improved version of BERT) to perform feature extraction on the word vector of each word in the video conversation data to obtain the first text modality feature;

[0033] The first feature is obtained based on the first text modality feature, the first video modality feature, and the first audio modality feature.

[0034] S2. Encode the first feature to obtain a second feature, where the second feature includes a second text modality feature, a second video modality feature, and a second audio modality feature. The specific steps include:

[0035] The first video modality feature and the first audio modality feature are encoded based on the normalization layer and the fully connected layer to obtain the second video modality feature and the second audio modality feature respectively. The first calculation formula for encoding may be:

[0036] U a =FC(Norm(X a ));

[0037] U v =FC(Norm(X v ));

[0038] Among them, U a represents the second audio modal feature, U v represents the second video modality feature, FC represents the fully connected layer, Norm represents the normalization layer, and X a represents the first audio modal feature, Xv represents a first video modality feature;

[0039] The first text modality feature is encoded based on the normalization layer and the bidirectional LSTM layer to obtain the second text modality feature. The second calculation formula for encoding can be:

[0040]

[0041] Among them, U t represents the second text modality feature, X t represents the first text modality feature, Norm represents the normalization layer, represents a bidirectional LSTM layer;

[0042] The second feature is obtained based on the second text modality feature, the second video modality feature and the second audio modality feature.

[0043] S3: constructing a cross-modal heterogeneous graph based on the second feature, wherein the cross-modal heterogeneous graph includes a first cross-modal heterogeneous graph, a second cross-modal heterogeneous graph, and a third cross-modal heterogeneous graph; the specific steps include:

[0044] Based on the second feature, obtaining a first node of a text mode, a second node of a video mode, and a third node of an audio mode;

[0045] A first node set and a first connection edge set are obtained based on the first node and the second node; and a first cross-modal heterogeneous graph is constructed based on the first node set and the first connection edge set. The first cross-modal heterogeneous graph can be expressed as G TV =(V tv , E tv ), V tv A set of nodes representing text mode and video mode, E tv A set of edges representing the connection between text modality nodes and video modality nodes;

[0046] A second node set and a second connection edge set are obtained based on the first node and the third node; and a second cross-modal heterogeneous graph is constructed based on the second node set and the second connection edge set. The second cross-modal heterogeneous graph can be expressed as G TA =(V ta , E ta ), V ta A collection of nodes representing text mode and audio mode, E ta A set of edges representing the connection between text modality nodes and audio modality nodes;

[0047] A third node set and a third connection edge set are obtained based on the second node and the third node; and a third cross-modal heterogeneous graph is constructed based on the third node set and the third connection edge set. The third cross-modal heterogeneous graph can be expressed as G AV =(V av , E av ), V av The node set E represents the audio mode and video mode in the graph. av A set of edges representing the connection between audio modality nodes and video modality nodes;

[0048] The cross-modal heterogeneous graph is obtained based on the first cross-modal heterogeneous graph, the second cross-modal heterogeneous graph, and the third cross-modal heterogeneous graph.

[0049] S4, adopting the collaborative attention mechanism, learning cross-modal interaction information based on the cross-modal heterogeneous graph, and obtaining attention weights, which include G TV The attention weight of the edge between the text modality and video modality nodes in the figure, G TA The attention weights of the edges between the text modality and audio modality nodes in the graph, and G AV The attention weights of the edges between the audio and video modality nodes in the figure; the calculation method of the attention weights of the edges between the text and audio modality nodes can be:

[0050]

[0051] Among them, ω t|a represents the attention weight of the audio modality to the text modality, ω a|t represents the attention weight of the text modality to the audio modality, SoftMax represents the activation function, and K a and K t Denote the key matrices of audio mode and text mode respectively, Q t and Q a They represent the query matrices of text mode and audio mode respectively, and d1 represents K a The dimension of d2 represents K t The dimension of , T represents transpose.

[0052] Similarly, referring to the above formula, the attention weights of the edges between the text modality and video modality nodes and the attention weights of the edges between the audio modality and video modality nodes are calculated.

[0053] S5. Updating the cross-modal heterogeneous graph based on the attention weight to obtain a final cross-modal heterogeneous graph; the specific steps include:

[0054] S501, obtaining neighbor nodes of the fourth node in the cross-modal heterogeneous graph, and aggregating the neighbor nodes based on the attention weights to obtain an aggregation result; for G TA Graph, aggregate the neighbor nodes of each node in the graph, such as the central node Aggregation method: The K neighbor nodes of calculate Aggregation features Its aggregation characteristics The calculation formula can be:

[0055]

[0056] in, Represents the central node The aggregate features of K neighbor nodes, represents the central node, express The set of K neighbor nodes of and All represent learnable parameter matrices, P represents the number of attention heads, p represents the pth attention head, represents the attention weight of the text modality of the p-th attention head for the audio modality, represents the attention weight of the audio modality of the p-th attention head for the text modality, Π represents the concatenation operation, and Respectively represent the discourse u m The characteristics of the text mode node and the audio mode node, V a Represents G TA The set of all audio nodes in the graph, V t Represents G TA The set of all text nodes in the graph.

[0057] Similarly, refer to the above formula to calculate G TV Figure and G AV Aggregate features of the graph to obtain aggregation results.

[0058] S502: Update the fourth node based on the aggregation result to obtain a fifth node; The feature representation of the central node is updated by combining its own features with the features of its neighbor set. The update calculation method can be:

[0059]

[0060] in, Represents the central node Updated features, Represents the central node The features before updating, W t∪a represents the learnable matrix, Represents the central node Aggregate features of K neighbor nodes.

[0061] S503: Update the cross-modal heterogeneous graph based on the fifth node to obtain the final cross-modal heterogeneous graph, that is, update the fourth node of the cross-modal heterogeneous graph to the fifth node to obtain the final cross-modal heterogeneous graph.

[0062] S6. Extracting complementary feature information of the final cross-modal heterogeneous graph; based on the complementary feature information, fusing features of the final cross-modal heterogeneous graph to obtain enhanced features;

[0063] The complementary feature information includes:

[0064] The video modality is converted into a first complementary feature of the text modality, and the text modality is converted into a second complementary feature of the video modality;

[0065] The third complementary feature of the audio modality is converted into the text modality, and the fourth complementary feature of the text modality is converted into the audio modality;

[0066] a fifth complementary feature of the conversion of the video modality to the audio modality, and a sixth complementary feature of the conversion of the audio modality to the video modality;

[0067] As for G TA Figure, central node Updated Features According to the mode they belong to, they are divided into and and Represent the text features of the fused audio and the audio features of the fused text, respectively. Similarly, in G TV Figure and G AV In the figure, by and Get the features respectively and and and Respectively represent G TV Figure and G AV The characteristics of the central node of the graph, and Respectively represent G TV The figure shows the text features of the fused video and the video features of the fused text. and Respectively represent G AVThe figure shows the audio features of the fused video and the video features of the fused audio.

[0068] The specific steps of obtaining the enhanced features include: based on the complementary feature information and the final cross-modal heterogeneous graph, the features of the nodes related to the same modality are fused to obtain the enhanced features. That is, each node in the updated graph receives the supplementary information of an additional modality, and the feature information of one modality contains information from the other two modalities. The features related to the same modality are fused to obtain more comprehensive complementary information, and the fusion calculation method can be:

[0069] S t =S t|v +S t|a ;

[0070] S v =S v|t +S v|a ;

[0071] S a =S a|v +S a|t ;

[0072] Among them, S t , S v and S a They represent the enhanced text modality features, video modality features and audio modality features respectively, and S t |v and S t|a They represent the text features of the fused video and the text features of the fused audio, respectively, and S v|t and S v|a They represent the video features of fused text and fused audio, respectively, and S a|v and S a|t They represent the audio features of the fused video and the audio features of the fused text respectively.

[0073] S7. Constructing a multimodal directed graph based on the enhanced features. The specific steps of constructing the multimodal directed graph include:

[0074] Based on the enhanced features, discourse nodes are obtained; based on the discourse nodes, discourse edges and relationship types of the discourse edges are obtained; the discourse nodes include text discourse nodes, video discourse nodes and audio discourse nodes; discourse edges include external edges between discourse nodes of different modalities and internal edges between discourse nodes of the same modality; the relationship types of the edges include external edge types and internal edge types; based on the time sequence of the discourse nodes, M past discourse nodes and N future discourse nodes of the discourse nodes are controlled to obtain the multimodal directed graph, where M and N both represent integers greater than or equal to 1.

[0075] For example, a multimodal directed graph is defined as: G = (V, E, R), where V represents the set of discourse nodes with three modal types, E represents the set of discourse edges, and R represents the set of relation types of discourse edges; each discourse node u i Include text discourse nodes Video Discourse Node and audio utterance nodes but η∈(v,t,a), v, t and a represent the video modality, text modality and audio modality respectively, r ij ∈R represents a node and nodes The relationship type of the edges between them, including the external edges R connecting different modal nodes inter Internal edges R connected to the same modal node intra , can be defined as:

[0076]

[0077] Different processing is performed on the interactions between nodes that occur in different time sequences in the graph, and M and N are used as sliding windows to control the current node The past M discourse nodes and the future N discourse nodes can be expressed as:

[0078]

[0079] in, and Represent the past and future relations respectively, i, j are both positive integers, u i and u j Both represent discourse nodes, R relation represents the structure of a multimodal directed graph, η∈(v,t,a), where v, t and a represent the video modality, text modality and audio modality respectively.

[0080] S8. Obtaining emotional features based on the multimodal directed graph; the specific steps of obtaining the emotional features include:

[0081] S801, extracting the dependency relationship between internal nodes in the discourse node and between adjacent discourse nodes through a relational graph convolution network (RGCN) to obtain a third feature;

[0082] Among them, the first calculation formula for obtaining the third feature is:

[0083]

[0084] in, represents the third feature of the i-th discourse node, ω top and ω tBoth represent learnable parameters, represents the features of the ith discourse node before the dependency is extracted, P t(i) Representation Node The set of neighbor indices of |P t(i) | represents the normalization constant, Representation Node The features of the neighbor nodes of , η∈(v,t,a), v, t and a represent the video modality, text modality and audio modality respectively, and i and m represent serial numbers.

[0085] S802. Based on the third feature, extract the local information and global information of the multimodal directed graph to obtain the fourth feature; for example, extract feature-rich representation through a graph attention network (GraphTransformer), each layer of which includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism allows vertices to use information from the neighborhood, and can capture local and global information in the graph at the same time. The calculation method can be:

[0086]

[0087] in, represents the fourth feature of the i-th discourse node, represents the third feature of the i-th discourse node, represents the third feature of the neighbor node of the i-th discourse node, W1 and W4 both represent learnable weight matrices for linear transformation, W2 and W3 both represent learnable weight matrices for calculating attention coefficients, d represents the dimension of the feature vector used for attention calculation, P (i) represents the set of all neighbor nodes of node i, softmax represents the activation function, η represents three different modalities, including text modality, video modality and audio modality, i and m both represent ordinal numbers, T represents transposition, || represents the connection operation, C represents the number of attention heads, c represents an integer greater than or equal to 1, η∈(v,t,a), v, t and a represent video modality, text modality and audio modality, respectively.

[0088] S803, using a graph sampling and aggregation network (GraphSAGE) to perform neighbor sampling and aggregation on the fourth feature to obtain the sentiment feature, and the sampling and aggregation calculation method can be:

[0089]

[0090] in, represents the local representation of all neighbor nodes of the ith discourse node in the lth layer network, represents the set of all neighbor nodes of the i-th discourse node, Represents the fourth feature of the i-th node, AGG(l) represents the aggregation function of the lth layer network, CONCAT represents concatenation, and W (l) represents the weight matrix of the lth layer network, σ represents the ReLU activation function, i represents the sequence number, represents the sentiment feature of the i-th discourse node in the l-th layer of the network, i and u both represent serial numbers, η∈(v,t,a), v, t and a represent video modality, text modality and audio modality respectively.

[0091] S9, transmitting the emotion feature to the pre-trained emotion classifier for prediction to obtain the predicted emotion result. If the emotion feature is input into the fully connected layer for prediction, the calculation method can be:

[0092] ρ i =softmax(fc(k i ));

[0093] Among them, ρ i represents the predicted emotion probability vector of the i-th emotion feature, softmax represents the activation function, fc represents the fully connected layer, k i represents the i-th sentiment feature, and i represents the sequence number.

[0094] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0095] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal conversation emotion recognition method based on cross-modal feature complementarity and directed graph learning, characterized in that: The method comprises: Acquire video conversation data, perform feature extraction on the video conversation data to obtain a first feature; encode the first feature to obtain a second feature; constructing a cross-modal heterogeneous graph based on the second feature, and obtaining an attention weight based on the cross-modal heterogeneous graph; updating the cross-modal heterogeneous graph based on the attention weight to obtain a final cross-modal heterogeneous graph, extracting complementary feature information of the final cross-modal heterogeneous graph, and fusing features of the final cross-modal heterogeneous graph based on the complementary feature information to obtain enhanced features; Constructing a multimodal directed graph based on the enhanced features; obtaining emotional features based on the multimodal directed graph, transmitting the emotional features to a pre-trained emotional classifier for prediction, and obtaining a predicted emotional result; The specific steps of constructing a multimodal directed graph include: Obtaining a discourse node based on the enhanced feature, obtaining a discourse edge and a relationship type of the discourse edge based on the discourse node, the discourse node including a text discourse node, a video discourse node and an audio discourse node, the discourse edge including an external edge between discourse nodes of different modalities and an internal edge between discourse nodes of the same modality, and the relationship type of the edge including an external edge type and an internal edge type; based on the time sequence of the discourse node, controlling M past discourse nodes and N future discourse nodes of the discourse node to obtain the multimodal directed graph, where M and N both represent integers greater than or equal to 1; The specific steps to obtain emotional features include: Extracting the dependency relationship between internal nodes in the discourse node and between adjacent discourse nodes to obtain a third feature; extracting local information and global information of the multimodal directed graph based on the third feature to obtain a fourth feature; performing neighbor sampling and aggregation on the fourth feature to obtain the sentiment feature; The first calculation formula for obtaining the third feature is: in, represents the third characteristic, ω top and ω t Both represent learnable parameters, Represents the characteristics of the discourse node, P t(i) Indicates that the node under the relation t∈R The set of neighbor indices of |P t(i) | represents the normalization constant, Representation Node The features of the neighbor nodes of , η represents three different modalities, including text modality, video modality and audio modality, and i and m both represent serial numbers.

2. According to claim 1, a multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning is characterized in that: The first feature includes a first text modality feature, a first video modality feature, and a first audio modality feature. The specific steps of obtaining the first feature include: Extracting features of an audio sequence in the video conversation data to obtain the first audio modality feature; Extracting features of facial expression changes and gesture changes in the video conversation data to obtain the first video modality features; Performing feature extraction on the word vector of each word in the video conversation data to obtain the first text modality feature; The first feature is obtained based on the first text modality feature, the first video modality feature, and the first audio modality feature.

3. According to claim 2, a multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning is characterized in that: The second feature includes a second text modality feature, a second video modality feature, and a second audio modality feature. The specific steps of obtaining the second feature include: Encoding the first video modality feature and the first audio modality feature based on a normalization layer and a fully connected layer to obtain the second video modality feature and the second audio modality feature respectively; Encoding the first text modality feature based on a normalization layer and a bidirectional LSTM layer to obtain the second text modality feature; The second feature is obtained based on the second text modality feature, the second video modality feature and the second audio modality feature.

4. According to claim 3, a multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning is characterized in that: The cross-modal heterogeneous graph includes a first cross-modal heterogeneous graph, a second cross-modal heterogeneous graph and a third cross-modal heterogeneous graph. The specific steps of constructing the cross-modal heterogeneous graph include: Based on the second feature, obtaining a first node of a text mode, a second node of a video mode, and a third node of an audio mode; Obtaining a first node set and a first connection edge set based on the first node and the second node; constructing the first cross-modal heterogeneous graph based on the first node set and the first connection edge set; Obtain a second node set and a second connection edge set based on the first node and the third node; construct the second cross-modal heterogeneous graph based on the second node set and the second connection edge set; Obtaining a third node set and a third connection edge set based on the second node and the third node; constructing the third cross-modal heterogeneous graph based on the third node set and the third connection edge set; The cross-modal heterogeneous graph is obtained based on the first cross-modal heterogeneous graph, the second cross-modal heterogeneous graph, and the third cross-modal heterogeneous graph.

5. According to claim 4, a multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning is characterized in that: The specific steps to obtain the final cross-modal heterogeneous graph include: Obtain neighbor nodes of a fourth node in the cross-modal heterogeneous graph, aggregate the neighbor nodes based on the attention weights to obtain an aggregation result, update the fourth node based on the aggregation result to obtain a fifth node; and update the cross-modal heterogeneous graph based on the fifth node to obtain the final cross-modal heterogeneous graph.

6. According to claim 5, a multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning is characterized in that: Complementary feature information includes: The video modality is converted into a first complementary feature of the text modality, and the text modality is converted into a second complementary feature of the video modality; The third complementary feature of the audio modality is converted into the text modality, and the fourth complementary feature of the text modality is converted into the audio modality; The video modality is converted into the fifth complementary feature of the audio modality, and the audio modality is converted into the sixth complementary feature of the video modality.

7. The multimodal dialogue emotion recognition method based on cross-modal feature complementation and directed graph learning according to claim 6 is characterized in that: The specific steps to obtain enhanced features include: Based on the complementary feature information and the final cross-modal heterogeneous graph, the features of nodes related to the same modality are fused to obtain the enhanced features.

Images