A multimodal emotion recognition method and device

Abstract

The present invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method and device, including: constructing a multimodal emotion recognition model; performing single-modal feature encoding on multimodal emotion dialogue data respectively; establishing a discourse dependency graph, a psychological dependency graph, and a semantic relationship graph for each modality, and sequentially performing graph convolution operations to obtain different types of predicted modal features for each modality; performing feature fusion on different types of predicted modal features belonging to the same modality, and inputting them into a multimodal attention layer in a multimodal feature fuser; and obtaining predicted emotion labels using an emotion classifier. The present invention realizes the establishment of multimodal contextual dependencies using a graph convolutional network by constructing multiple types of dependency graphs, better capturing the rich relationships between different modalities and within the same modality, and improving the use of associated information by the multimodal emotion recognition model, thereby improving the accuracy of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method and device. Background Art

[0002] In recent years, with the widespread application of multimodal data, multimodal emotion recognition (MERMC) has become a hot research direction in the field of artificial intelligence. MERMC aims to automatically recognize and understand human emotions by analyzing data in multiple modalities such as text, audio, and vision.

[0003] Traditional emotion recognition methods mainly focus on single-modality emotion analysis, such as using only text or audio for emotion analysis. Although these methods have achieved some success in specific modalities, they fail to fully utilize the information of multimodal data.

[0004] However, multimodal data in actual scenarios can provide richer information, which has attracted widespread attention from technicians to MERMC. Some existing technologies attempt to simply splice or weightedly fuse features from different modalities through simple modal fusion methods, such as tensor fusion networks, gating mechanisms, and combination strategies. However, this simple fusion method has certain limitations in complex and unstructured multi-party conversations and cannot capture diverse contextual dependencies. Some existing technologies have explored the application of graph convolutional networks (GCN) in multimodal emotion recognition, usually using graph convolutional networks to establish contextual dependencies of a single modality, but cannot effectively solve the interactivity and consistency problems in multimodal data. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that it is impossible to solve the interactivity and consistency in multimodal data while considering the contextual dependencies in the multimodal data.

[0006] In order to solve the above technical problems, the present invention provides a multimodal emotion recognition method, comprising:

[0007] Constructing a multimodal emotion recognition model, wherein the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fuser and an emotion classifier;

[0008] Collect multimodal emotional dialogue data including textual modality, audio modality and visual modality, input them into the modality extractor of the multimodal emotion recognition model, encode the multimodal emotional dialogue data into single modal features, and obtain textual feature sequences, audio feature sequences and visual feature sequences;

[0009] The text feature sequence, audio feature sequence and visual feature sequence are input into the multivariate dependency encoder to establish the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality, including:

[0010] Structural modeling is performed on the text feature sequence, and each modality constructs a discourse dependency graph of each modality based on the results of structural modeling;

[0011] Speaker modeling is performed on the text feature sequence, and each modality constructs a psychological dependency graph of each modality based on the results of speaker modeling;

[0012] Semantic modeling is performed on the text feature sequence, audio feature sequence and visual feature sequence respectively, and a semantic relationship graph of each modality is constructed according to its corresponding semantic modeling result;

[0013] The discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality are input into the multimodal feature fuser, and graph convolution operations are performed in sequence to obtain different types of prediction modal features of each modality; different types of prediction modal features belonging to the same modality are subjected to feature fusion to obtain text modal fusion features, audio modal fusion features and visual modal fusion features; the text modal fusion features are used as query features, the audio modal fusion features are used as key features, and the visual modal fusion features are used as value features, and are input into the multimodal attention layer in the multimodal feature fuser;

[0014] The output features of the multimodal attention layer are input into the sentiment classifier to obtain the predicted sentiment label.

[0015] In one embodiment of the present invention, the modal extractor includes a text feature extractor, an audio feature extractor and a visual feature extractor; the text feature extractor adopts text embedding technology, including a pre-trained word embedding model or a Transformer model; the audio feature extractor adopts an acoustic feature extraction method, including a Mel spectrogram or a Mel-frequency cepstral coefficient; the visual feature extractor includes a pre-trained convolutional neural network model.

[0016] In one embodiment of the present invention, the step of inputting the text feature sequence, the audio feature sequence and the visual feature sequence into a multivariate dependency encoder to establish a discourse dependency graph, a psychological dependency graph and a semantic relationship graph of each modality includes:

[0017] The structure modeling is performed based on the text feature sequence, and the pre-trained conversation parser is used to determine the relationship between the current discourse and the historical discourse in the text modality, and the triple cu is obtained. i ,r ij ,u j >, in order to represent the interaction between different node discourses, where u i and u j denote the utterances of node i and node j respectively, rij Represents the structural relationship between the discourse of node i and node j; the triples obtained by each modality based on structural modeling i ,r ij ,u j >Construct a discourse dependency graph for each modality, where the edge connections and edge relationships between different nodes in the discourse dependency graph are based on the structural relationship between the corresponding nodes r ij Establish;

[0018] Speaker modeling is performed based on the text feature sequence, and the social common sense knowledge base is used to determine the intention or reaction representation of the speaker's behavior in the current discourse under the text modality to obtain a triple i ,r' ij ,u j >, in order to represent the interaction between different node speakers, where r' ij Represents the psychological relationship between the speakers of node i and node j; the triples of each modality obtained by modeling the speakers i ,r' ij ,u j >Construct a psychological dependency graph for each modality, where the edge connections and edge relationships between different nodes in the psychological dependency graph are based on the psychological relationships r' between the corresponding nodes ij Establish;

[0019] Semantic modeling is performed on each modality based on the text feature sequence, audio feature sequence and visual feature sequence, and the cosine similarity is used to measure the semantic similarity between the discourse semantics of different nodes in the same modality to obtain a semantic similarity score; a similarity matrix of each modality is constructed based on the semantic similarity score between the discourse of the current node of each modality and the discourse of other nodes in the conversation; a semantic relationship graph of each modality is constructed based on the similarity matrix obtained by the corresponding semantic modeling of each modality, and the edge connections and edge relationships between different nodes in the semantic relationship graph are established based on the semantic similarity scores between the corresponding nodes in the similarity matrix.

[0020] In one embodiment of the present invention, the formula for the semantic similarity score is:

[0021]

[0022] Among them, m is the modality, including audio modality a, visual modality v and text modality t, is the semantic similarity score between node i and node j in mode m, is the eigenvector of node i in mode m, is the eigenvector of node j under mode m, arccos is the inverse cosine function, is the feature vector The L2 norm of is the feature vector ​​​The L2 norm of .

[0023] In one embodiment of the present invention, the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality are input into the multimodal feature fusion device, and graph convolution operations are performed in sequence to obtain different types of predicted modal features of each modality. The formula is:

[0024]

[0025] in, is the predicted modal feature of node i with modeling type x under modality m, x is the modeling type, including structural modeling, speaker modeling and semantic modeling, m is the modality, including audio modality a, visual modality v and text modality t; σ is the Sigmoid activation function; Model the hidden representation of node i of type x under modality m; for The weight matrix of m It means that r belongs to one of the modes m; A set of neighbor nodes of node i, where the neighbor nodes are other nodes that are in the same dialogue with node i; is the normalization constant, Model the hidden representation of the neighbor nodes of type x node i under modality m, for The weight matrix of .

[0026] In one embodiment of the present invention, the prediction modal features of different types belonging to the same modality are fused to obtain text modal fusion features, audio modal fusion features, and visual modal fusion features, and the formula is:

[0027]

[0028] in, is the fusion feature of modality m of node i, where m is the modality, including audio modality a, visual modality v and text modality t; is the fusion feature of the structural modeling node i under mode m, is the fusion feature of the speaker modeling node i under modality m, is the fusion feature of the semantic modeling node i under modality m, Represents a concatenation operation.

[0029] In one embodiment of the present invention, the text modality feature is used as the query feature, the audio modality feature is used as the key feature, and the visual modality feature is used as the value feature, which is input into the multimodal attention layer in the multimodal feature fuser, and the formula includes:

[0030]

[0031] e i =MMA(Q i ,K i ,V i )

[0032] Among them, Q i is the query feature of node i, is the text modality fusion feature of node i, is the query weight matrix of node i; K i is the key feature of node i, is the audio modality fusion feature of node i, is the key weight matrix of node i; V i is the value feature of node i, is the visual modality fusion feature of node i, is the value weight matrix of node i; MMA is the multimodal attention layer, e i is the output feature of node i in the multimodal attention layer.

[0033] In one embodiment of the present invention, the output features of the multimodal attention layer are input into a sentiment classifier to obtain a predicted sentiment label, and the formula includes:

[0034] e' i =ReLU(W rl e i +b rl )

[0035] P i =softmax(W smax e′ i +b smax )

[0036]

[0037] Among them, e i is the output feature of node i in the multimodal attention layer, ReLU(·) is the ReLU activation function, W rl is the weight matrix of ReLU activation, b rl is the bias term of ReLU activation, e' i is the output feature of node i in the ReLU activation function; softmax(·) is the softmax function, W smax is the weight matrix of the softmax function, b smax is the bias term of the softmax function, P i is the probability distribution of the predicted sentiment label of node i, λ is P i Take the maximum subscript value; is the predicted sentiment label of node i.

[0038] In one embodiment of the present invention, the loss function of the multimodal emotion recognition model is:

[0039]

[0040] Where L is the loss function of the multimodal emotion recognition model, S is the total number of dialogues, Z(s) is the number of nodes in dialogue s, and P s,z is the predicted sentiment label of node z in dialogue s, is the cross entropy term in the loss function, which represents the difference between the predicted probability distribution of the sentiment label and the true label, e s,z is the output feature of node z in the multimodal attention layer in dialogue s, η||θ|| 2 is the L2 regularization term used to control the complexity of the model, and η and θ are the hyperparameters of the regularizer.

[0041] The present invention also provides a multimodal emotion recognition device, comprising:

[0042] A model building module, used to build a multimodal emotion recognition model, wherein the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fuser and an emotion classifier;

[0043] A feature extraction module is used to collect multimodal emotional dialogue data including textual modality, audio modality and visual modality, input the data into the modality extractor of the multimodal emotion recognition model, encode the multimodal emotional dialogue data into single modal features, and obtain textual feature sequences, audio feature sequences and visual feature sequences;

[0044] A multivariate dependency graph generation module is used to input a text feature sequence, an audio feature sequence, and a visual feature sequence into a multivariate dependency encoder to establish a discourse dependency graph, a psychological dependency graph, and a semantic relationship graph of each modality, including: structural modeling of the text feature sequence, and each modality constructs a discourse dependency graph of each modality according to the result of the structural modeling; speaker modeling of the text feature sequence, and each modality constructs a psychological dependency graph of each modality according to the result of the speaker modeling; semantic modeling of the text feature sequence, the audio feature sequence, and the visual feature sequence is performed respectively, and each modality constructs a semantic relationship graph of each modality according to its corresponding semantic modeling result;

[0045] The cross-modal fusion module is used to input the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality into the multimodal feature fuser, and perform graph convolution operations in sequence to obtain different types of prediction modal features of each modality; perform feature fusion on different types of prediction modal features belonging to the same modality to obtain text modal fusion features, audio modal fusion features, and visual modal fusion features; use the text modal fusion features as query features, the audio modal fusion features as key features, and the visual modal fusion features as value features, and input them into the multimodal attention layer in the multimodal feature fuser;

[0046] The classification module is used to input the output features of the multimodal attention layer into the sentiment classifier to obtain the predicted sentiment label.

[0047] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0048] A multimodal emotion recognition method described in the present invention extracts feature information of multiple modalities of text, audio and vision in dialogue data, and constructs a multimodal emotion recognition model to recognize the emotion information in the dialogue data. The multivariate dependency encoder of the multimodal emotion recognition model performs structural modeling, speaker modeling and semantic modeling on text features, audio features and visual features, establishes discourse dependency graphs, psychological dependency graphs and semantic relationship graphs of each modality, better captures the rich relationships within the same modality, and improves the comprehensiveness and accuracy of the relationship recognition between different modalities. The multimodal feature fusion device of the multimodal emotion recognition model uses a graph convolutional network to propagate and aggregate information in multiple dependency graphs, so that each node can contain contextual information from other nodes, which helps to better understand the context and psychological state of the entire dialogue, thereby improving the consistency and relevance of information between modalities. And by introducing a multimodal attention mechanism, the model can adaptively focus on more important information in different modalities, which helps to improve the attention of emotional information in the dialogue, thereby improving the performance of emotion classification and improving the accuracy of emotion classification.

[0049] In summary, the present invention constructs multiple types of dependency graphs and realizes the establishment of multimodal contextual dependency relationships using graph convolutional networks, better captures the rich relationships between different modalities and within the same modality, improves the utilization effect of associated information by multimodal emotion recognition models, and thus improves the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0051] Figure 1 is a flow chart of a multimodal emotion recognition method of the present invention;

[0052] Figure 2 is a structural diagram of the multimodal emotion recognition model of the present invention;

[0053] Figure 3 This is a comparison chart between the effects of a multimodal emotion recognition method of the present invention and the prior art. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0055] Reference Figure 1 As shown, the present invention provides a multimodal emotion recognition method, which recognizes emotions in multimodal emotion dialogue data by constructing a multimodal emotion recognition model. Figure 2 As shown, the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fusion device and an emotion classifier. The specific steps include:

[0056] Collect multimodal emotional dialogue data including text modality, audio modality and visual modality to ensure that the dataset is diverse and covers different scenarios, speaking styles and emotional expressions.

[0057] The multimodal emotional dialogue data has already had nodes divided in the data set, and the emotional category is marked for each node, and the marking is ensured to be accurate and consistent, so as to establish a training goal for emotional classification.

[0058] The text in the multimodal emotional dialogue data is preprocessed by word segmentation, stop word removal, word embedding, etc.

[0059] The pre-processed text, audio and visual data are input into the modal extractor of the multimodal emotion recognition model, and the multimodal emotional dialogue data are respectively encoded with single modal features. The modal extractor includes a text feature extractor, an audio feature extractor and a visual feature extractor.

[0060] The text feature extractor uses text embedding technology, including pre-trained word embedding models such as Word2Vec, GloVe, or Transformer models such as BERT, GPT, to map each word to a high-dimensional vector representation.

[0061] The audio feature extractor uses an acoustic feature extraction method, including a Mel spectrogram or Mel frequency cepstral coefficients (MFCC), to capture the spectral information of the audio signal and provide the model with information about the audio content.

[0062] The visual feature extractor uses a pre-trained convolutional neural network (CNN) to extract features of an image. Models such as ResNet and Inception can be used to extract high-level features in an image, and these features can be used as representations of visual modalities.

[0063] Taking the text mode as an example, the text data is preprocessed, and the formula is:

[0064]

[0065] Among them, [CLS] is a special identifier, which is appended to the beginning of the sentence to generate the features of the whole sentence. is the original text data of the ith node, U T Enter a sequence for the text;

[0066] Enter the text into the sequence U T Input to the text extractor, the formula is:

[0067] H T =TextEncoder(U T )

[0068] Among them, TextEncoder is a text extractor, H T is a sequence of text features.

[0069] If the text feature sequence, audio feature sequence and visual feature sequence output by the modal extractor are directly combined into a graph structure, these graphs can capture the relationship between sentences, but are still limited by the inherent protocol of graph learning, that is, the integration of local neighborhood messages for propagation prevents the context and speaker information interaction between sentences in the conversation. To this end, the present invention uses a multivariate dependency encoder for the text feature sequence, audio feature sequence and visual feature sequence to establish a discourse dependency graph, a psychological dependency graph and a semantic relationship graph for each modality, specifically including:

[0070] Structural modeling is performed based on the text feature sequence, and the pre-trained conversation parser is used to determine the relationship between the current discourse and the historical discourse in the text modality to obtain the triple i ,r ij ,u j >, in order to represent the interaction between different node discourses, where u i and u j denote the utterances of node i and node j respectively, r ij Represents the structural relationship between the discourse of node i and node j; the triples obtained by each modality based on structural modeling i ,r ij ,u j ​​>Construct a discourse dependency graph for each modality, where the edge connections and edge relationships between different nodes in the discourse dependency graph are based on the structural relationship between the corresponding nodes r ij Established to construct directed graphs in different modes.

[0071] Speaker modeling is performed based on the text feature sequence, and the social common sense knowledge base is used to determine the intention or reaction representation of the speaker's behavior in the current discourse under the text modality to obtain a triple i ,r' ij ,u j >, in order to represent the interaction between different node speakers, where r' ij Represents the psychological relationship between the speakers of node i and node j; the triples of each modality obtained by modeling the speakers i ,r' ij ,u j >Construct a psychological dependency graph for each modality, where the edge connections and edge relationships between different nodes in the psychological dependency graph are based on the psychological relationships r' between the corresponding nodes ij Established to construct directed graphs in different modes.

[0072] Semantic modeling is performed on each modality based on the text feature sequence, audio feature sequence and visual feature sequence, and the cosine similarity is used to measure the similarity between the semantics of discourses of different nodes in the same modality to obtain a semantic similarity score; a similarity matrix of each modality is constructed based on the semantic similarity score between the discourse of the current node of each modality and the discourse of other nodes in the conversation; a semantic relationship graph of each modality is constructed based on the similarity matrix obtained by the corresponding semantic modeling of each modality, and the edge connections and edge relationships between different nodes in the semantic relationship graph are established based on the semantic similarity scores between the corresponding nodes in the similarity matrix, so as to construct a directed graph under different modalities.

[0073] The formula for calculating the semantic similarity score is:

[0074]

[0075] Among them, m is the modality, including audio modality a, visual modality v and text modality t, is the semantic similarity score between node i and node j in mode m, is the eigenvector of node i in mode m, is the eigenvector of node j under mode m, arccos is the inverse cosine function, is the feature vector The L2 norm of is the feature vector The L2 norm of .

[0076] ​​The multivariate dependency encoder utilizes the diversity of context dependencies within each modality to perform context modeling, adopts structural modeling to capture the deep structural representation between different discourses, adopts speaker modeling to capture the psychological activities between different speakers, and adopts semantic modeling to capture the connection between the semantics of different discourses, thereby providing multivariate and complex dependency relationships in multimodal emotional dialogue data.

[0077] Since information fusion between different modalities is a complex issue, in order to ensure the semantic consistency of modal representation across multiple modes in cross-modal representation learning, the present invention proposes a multimodal feature fuser for transferring prior information captured from the text modality to the audio and visual modalities, effectively and efficiently fusing the information of each modality and maintaining the semantic consistency of different modalities.

[0078] Since traditional deep learning models perform poorly when processing multimodal data with non-Euclidean structures, the present invention introduces a graph convolutional network, which helps to better process the non-Euclidean structure of multimodal data, thereby improving the multimodal emotion recognition model's ability to model complex relationships.

[0079] The discourse dependency graph, psychological dependency graph, and semantic relationship graph of each modality are input into the multimodal feature fusion device, and graph convolution operations are performed in sequence to aggregate and propagate various dependencies between sentence nodes to obtain different types of predicted modal features of each modality. The formula is:

[0080]

[0081] in, is the predicted modal feature of node i with modeling type x under modality m, x is the modeling type, including structural modeling, speaker modeling and semantic modeling, m is the modality, including audio modality a, visual modality v and text modality t; σ is the Sigmoid activation function; Model the hidden representation of node i of type x under modality m; for The weight matrix of m It means that r belongs to one of the modes m; A set of neighbor nodes of node i, where the neighbor nodes are other nodes that are in the same dialogue with node i; is the normalization constant, Model the hidden representation of the neighbor nodes of type x node i under modality m, for The weight matrix of .

[0082] After the graph convolution network, nine prediction modal features are obtained, which are the prediction modal features of the structural modeling node i in the text modality. Predicted modal features of the modeling type of speaker modeling node i in text mode Predicted modal features of semantic modeling node i in text mode Predicted modal features of the structural modeling node i in the audio mode Predicted modal features of speaker modeling node i in audio mode Predicted modal features of semantic modeling node i in audio modality Predicted modal features of structural modeling node i in visual mode Predicted modal features of the modeling node i with speaker modeling type in the visual modality Predicted modal features of semantic modeling node i in visual modality

[0083] The present invention utilizes graph convolutional networks for information aggregation and propagation, and its output contains contextual information and dependencies in the conversation, which helps to ensure that cross-modal relationships are considered for each sentence node in the multivariate dependency graph, thereby more comprehensively understanding the contextual relationships of emotions.

[0084] The prediction modal features of different types belonging to the same modality are fused to obtain the text modality fusion feature Audio modality fusion features Visual modality fusion features The formula is:

[0085]

[0086] in, is the fusion feature of modality m of node i, where m is the modality, including audio modality a, visual modality v and text modality t; is the fusion feature of the structural modeling node i under mode m, is the fusion feature of the speaker modeling node i under modality m, is the fusion feature of the semantic modeling node i under modality m, Represents a concatenation operation.

[0087] In order to enable the multimodal emotion recognition model to adaptively focus on more important information in different modalities, the present invention introduces a multimodal attention mechanism, that is, the text modality fusion feature is used as the query feature, the audio modality fusion feature is used as the key feature, and the visual modality fusion feature is used as the value feature, which is input into the multimodal attention layer in the multimodal feature fuser. The formula includes:

[0088]

[0089] e i=MMA(Q i ,K i ,V i )

[0090] Among them, Q i is the query feature of node i, is the text modality fusion feature of node i, is the query weight matrix of node i; K i is the key feature of node i, is the audio modality fusion feature of node i, is the key weight matrix of node i; V i is the value feature of node i, is the visual modality fusion feature of node i, is the value weight matrix of node i; MMA is the multimodal attention layer, e i is the output feature of node i in the multimodal attention layer.

[0091] The multimodal attention layer helps to increase the attention to the emotional information in the conversation, thereby improving the sentiment classification performance. The output features of the multimodal attention layer contain the fusion representation of the multimodal information in the conversation.

[0092] The output features of the multimodal attention layer are input into the sentiment classifier of the multimodal sentiment recognition model to obtain the predicted sentiment label.

[0093] The sentiment classifier includes multiple fully connected layers and a Softmax output layer. The fully connected layer is a densely connected neural network layer, the purpose of which is to map feature vectors to the space of sentiment labels by learning weights and biases. An activation function is usually added after the fully connected layer to introduce nonlinear relationships, such as ReLU (Rectified Linear Unit) or Sigmoid activation function, which helps the model learn more complex feature representations. Finally, the probability of multi-category sentiment labels is normalized through the Softmax layer. The Softmax function converts the original score output by the model into a probability distribution, so that the probability of each sentiment label is between 0 and 1, and the sum of the probabilities is 1.

[0094] The specific formula in the sentiment classifier includes:

[0095] e' i =ReLU(W rl e i +b rl )

[0096] P i =softmax(W smax e′ i +b smax)

[0097]

[0098] Among them, e i is the output feature of node i in the multimodal attention layer, ReLU(·) is the ReLU activation function, W rl is the weight matrix of ReLU activation, b rl is the bias term of ReLU activation, e' i is the output feature of node i in the ReLU activation function; softmax(·) is the softmax function, W smax is the weight matrix of the softmax function, b smax is the bias term of the softmax function, P i is the probability distribution of the predicted sentiment label of node i, λ is P i Take the maximum subscript value; is the predicted sentiment label of node i.

[0099] The training process of the multimodal emotion recognition model includes:

[0100] The multimodal emotional dialogue data is used as a dataset and divided into training set, validation set and test set, and the data in each set is ensured to be evenly distributed to avoid model overfitting.

[0101] Initialize the weights of the multimodal emotion recognition model and choose an appropriate initialization strategy to accelerate convergence.

[0102] Select an appropriate loss function: The present invention selects the cross entropy loss function commonly used in sentiment classification tasks as the loss function, and the formula is:

[0103]

[0104] Where L is the loss function of the emotion recognition model, S is the total number of dialogues, Z(s) is the number of nodes in dialogue s, and P s,z is the predicted sentiment label of node z in dialogue s, is the cross entropy term in the loss function, which represents the difference between the predicted probability distribution of the sentiment label and the true label, e s,z is the output feature of node z in the multimodal attention layer in dialogue s, η||θ|| 2 is the L2 regularization term used to control the complexity of the model, and η and θ are the hyperparameters of the regularizer.

[0105] Choose an appropriate optimizer, such as the Adam optimizer, to efficiently tune model parameters.

[0106] Feed the training set into the multimodal emotion recognition model, making sure the format of the data input matches the format expected by the model.

[0107] Forward propagation and back propagation are performed on the training data, and the model parameters are updated by gradient descent to ensure that the loss function is minimized on the training set.

[0108] Use the learning rate scheduler to dynamically adjust the learning rate according to the progress of training to improve the training effect.

[0109] The validation set is used during the training process to monitor the performance of the multimodal emotion recognition model, and the model is evaluated on the validation set to prevent overfitting.

[0110] Periodically save the model's weights so you can restore or continue training when needed.

[0111] Based on the performance of the validation set, adjust the model's hyperparameters, such as hidden layer size, learning rate, etc.

[0112] When the model achieves satisfactory performance on the validation set, training is terminated.

[0113] The test set is used to evaluate the performance of the final trained model to ensure that the multimodal emotion recognition model can generalize effectively on unseen data.

[0114] Reference Figure 3 As shown in the effect comparison diagram of the present invention and the prior art, a multimodal emotion recognition method described in the present invention extracts feature information of multiple modalities of text, audio and vision in the dialogue data, and constructs a multimodal emotion recognition model to identify the emotion information in the dialogue data. The multivariate dependency encoder of the multimodal emotion recognition model performs structural modeling, speaker modeling and semantic modeling on text features, audio features and visual features, establishes discourse dependency graphs, psychological dependency graphs and semantic relationship graphs of each modality, better captures the rich relationships within the same modality, and improves the comprehensiveness and accuracy of the relationship recognition between different modalities. The multimodal feature fusion device of the multimodal emotion recognition model uses a graph convolutional network to propagate and aggregate information in multiple dependency graphs, so that each node can contain contextual information from other nodes, which helps to better understand the context and psychological state of the entire dialogue, thereby improving the consistency and relevance of information between modalities. And by introducing a multimodal attention mechanism, the model can adaptively focus on more important information in different modalities, which helps to improve the attention of emotional information in the dialogue, thereby improving the performance of emotion classification and the accuracy of emotion classification.

[0115] In summary, the present invention constructs multiple types of dependency graphs and realizes the establishment of multimodal contextual dependencies using graph convolutional networks, better captures the rich relationships between different modalities and within the same modality, improves the utilization of associated information by multimodal emotion recognition models, and thus improves the accuracy of emotion recognition.

[0116] The present invention also provides a multimodal emotion recognition device, comprising:

[0117] A model building module, used to build a multimodal emotion recognition model, wherein the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fuser and an emotion classifier;

[0118] A feature extraction module is used to collect multimodal emotional dialogue data including textual modality, audio modality and visual modality, input the data into the modality extractor of the multimodal emotion recognition model, encode the multimodal emotional dialogue data into single modal features, and obtain textual feature sequences, audio feature sequences and visual feature sequences;

[0119] A multivariate dependency graph generation module is used to input a text feature sequence, an audio feature sequence, and a visual feature sequence into a multivariate dependency encoder to establish a discourse dependency graph, a psychological dependency graph, and a semantic relationship graph of each modality, including: structural modeling of the text feature sequence, and each modality constructs a discourse dependency graph of each modality according to the result of the structural modeling; speaker modeling of the text feature sequence, and each modality constructs a psychological dependency graph of each modality according to the result of the speaker modeling; semantic modeling of the text feature sequence, the audio feature sequence, and the visual feature sequence is performed respectively, and each modality constructs a semantic relationship graph of each modality according to its corresponding semantic modeling result;

[0120] The cross-modal fusion module is used to input the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality into the multimodal feature fuser, and perform graph convolution operations in sequence to obtain different types of prediction modal features of each modality; perform feature fusion on different types of prediction modal features belonging to the same modality to obtain text modal fusion features, audio modal fusion features, and visual modal fusion features; use the text modal fusion features as query features, the audio modal fusion features as key features, and the visual modal fusion features as value features, and input them into the multimodal attention layer in the multimodal feature fuser;

[0121] The classification module is used to input the output features of the multimodal attention layer into the sentiment classifier to obtain the predicted sentiment label. It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0125] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A multimodal emotion recognition method, characterized in that: include: Constructing a multimodal emotion recognition model, wherein the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fuser and an emotion classifier; Collect multimodal emotional dialogue data including textual modality, audio modality and visual modality, input them into the modality extractor of the multimodal emotion recognition model, encode the multimodal emotional dialogue data into single modal features, and obtain textual feature sequences, audio feature sequences and visual feature sequences; The text feature sequence, audio feature sequence and visual feature sequence are input into the multivariate dependency encoder to establish the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality, including: Structural modeling is performed based on the text feature sequence, and the pre-trained conversation parser is used to determine the relationship between the current discourse and the historical discourse in the text modality to obtain the triple , in order to represent the interaction between different node discourses, where and Respectively represent nodes and nodes The words, Representation Node and nodes The structural relationship between the discourses; the triples of each modality obtained according to the structural modeling Construct a discourse dependency graph for each modality, where the edge connections and edge relationships between different nodes in the discourse dependency graph are based on the structural relationships between the corresponding nodes. Establish; Speaker modeling is performed based on the text feature sequence, and the social common sense knowledge base is used to determine the intention or reaction representation of the speaker's behavior in the current discourse under the text modality to obtain a triple , in order to represent the interaction between different node speakers, where Representation Node and nodes The psychological relationship between the speakers; the triples of each modality obtained by modeling the speakers Construct a psychological dependency graph for each modality, wherein the edge connections and edge relationships between different nodes in the psychological dependency graph are based on the psychological relationships between the corresponding nodes. Establish; Semantic modeling is performed on each modality according to the text feature sequence, audio feature sequence and visual feature sequence, and the semantic similarity between the discourse semantics of different nodes in the same modality is measured by cosine similarity to obtain a semantic similarity score; a similarity matrix of each modality is constructed according to the semantic similarity score between the discourse of the current node of each modality and the discourse of other nodes in the conversation; a semantic relationship graph of each modality is constructed according to the similarity matrix obtained by the corresponding semantic modeling of each modality, and the edge connections and edge relationships between different nodes in the semantic relationship graph are established according to the semantic similarity scores between the corresponding nodes in the similarity matrix; The discourse dependency graph, psychological dependency graph, and semantic relationship graph of each modality are input into the multimodal feature fusion device, and graph convolution operations are performed in sequence to obtain different types of predicted modal features of each modality. The formula is: ； in, For modal The following modeling types are node The predicted modal features of Modeling types include structural modeling, speaker modeling, and semantic modeling. For modalities, including audio modalities , Visual Mode and text modal ; is the Sigmoid activation function; For modal The following modeling types are node Hidden representation of ; for The weight matrix of express Belong to modal one; With Node The set of neighbor nodes of node Other nodes that belong to the same conversation; is the normalization constant, For modal The following modeling types are node The hidden representation of the neighbor nodes of for The weight matrix of The features of different types of predicted modal features belonging to the same modality are fused to obtain text modal fusion features, audio modal fusion features, and visual modal fusion features. The formula is: ； in, For Node Modal The fusion characteristics of For modalities, including audio modalities , Visual Mode and text modal ; For modal The modeling type is the structural modeling node The fusion characteristics of For modal The modeling type is speaker modeling node The fusion characteristics of For modal The modeling type is semantic modeling node The fusion characteristics of Represents a splicing operation; The text modality fusion feature is used as the query feature, the audio modality fusion feature is used as the key feature, and the visual modality fusion feature is used as the value feature, and is input into the multimodal attention layer in the multimodal feature fuser; The output features of the multimodal attention layer are input into the sentiment classifier to obtain the predicted sentiment label.

2. A multimodal emotion recognition method according to claim 1, characterized in that: The modality extractor includes a text feature extractor, an audio feature extractor and a visual feature extractor; the text feature extractor adopts text embedding technology, including a pre-trained word embedding model or a Transformer model; the audio feature extractor adopts an acoustic feature extraction method, including a Mel spectrum or a Mel frequency cepstral coefficient; the visual feature extractor includes a pre-trained convolutional neural network model.

3. The multimodal emotion recognition method according to claim 1, characterized in that: The formula for the semantic similarity score is: ； in, For modalities, including audio modalities , Visual Mode and text modal , for Modal node and nodes The semantic similarity score between for Modal node The characteristic vector of for Modal node The characteristic vector of is the arccosine function, is the feature vector The L2 norm of is the feature vector The L2 norm of .

4. The multimodal emotion recognition method according to claim 1, characterized in that: The text modality fusion feature is used as the query feature, the audio modality fusion feature is used as the key feature, and the visual modality fusion feature is used as the value feature, which is input into the multimodal attention layer in the multimodal feature fuser. The formula includes: ； ； in, For Node The query features, For Node The text modality fusion features of For Node The query weight matrix of For Node The key features of For Node The audio modality fusion features of For Node The key weight matrix of For Node The value characteristics of For Node The visual modality fusion features of For Node The value weight matrix of is the multimodal attention layer, For Node Output features at the multimodal attention layer.

5. The multimodal emotion recognition method according to claim 1, characterized in that: The output features of the multimodal attention layer are input into the sentiment classifier to obtain the predicted sentiment label. The formula includes: ； ； ； in, For Node The output features of the multimodal attention layer, is the ReLU activation function, is the weight matrix of ReLU activation, is the bias term of ReLU activation, For Node Output features of the ReLU activation function; is the softmax function, is the weight matrix of the softmax function, is the bias term of the softmax function, For Node The probability distribution of predicted sentiment labels, for Take the maximum subscript value; For Node The predicted sentiment labels.

6. A multimodal emotion recognition method according to claim 1, characterized in that: The loss function of the multimodal emotion recognition model is: ； in, is the loss function of the multimodal emotion recognition model, is the total number of conversations, For dialogue The number of nodes in For dialogue Midpoint The predicted sentiment label of is the cross entropy term in the loss function, which represents the difference between the predicted probability distribution of the sentiment label and the true label. For dialogue Midpoint The output features of the multimodal attention layer, is the L2 regularization term, which is used to control the complexity of the model. and is the hyperparameter of the regularizer.

7. A multimodal emotion recognition device, characterized in that: include: A model building module, used to build a multimodal emotion recognition model, wherein the structure of the multimodal emotion recognition model includes a modality extractor, a multivariate dependency encoder, a multimodal feature fuser and an emotion classifier; A feature extraction module is used to collect multimodal emotional dialogue data including textual modality, audio modality and visual modality, input the data into the modality extractor of the multimodal emotion recognition model, encode the multimodal emotional dialogue data into single modal features, and obtain textual feature sequences, audio feature sequences and visual feature sequences; The multivariate dependency graph generation module is used to input the text feature sequence, audio feature sequence and visual feature sequence into the multivariate dependency encoder to establish the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality, including: Structural modeling is performed based on the text feature sequence, and the pre-trained conversation parser is used to determine the relationship between the current discourse and the historical discourse in the text modality to obtain the triple , in order to represent the interaction between different node discourses, where and Respectively represent nodes and nodes The words, Representation Node and nodes The structural relationship between the discourses; the triples of each modality obtained according to the structural modeling Construct a discourse dependency graph for each modality, where the edge connections and edge relationships between different nodes in the discourse dependency graph are based on the structural relationships between the corresponding nodes. Establish; Speaker modeling is performed based on the text feature sequence, and the social common sense knowledge base is used to determine the intention or reaction representation of the speaker's behavior in the current discourse under the text modality to obtain a triple , in order to represent the interaction between different node speakers, where Representation Node and nodes The psychological relationship between the speakers; the triples of each modality obtained by modeling the speakers Construct a psychological dependency graph for each modality, wherein the edge connections and edge relationships between different nodes in the psychological dependency graph are based on the psychological relationships between the corresponding nodes. Establish; Semantic modeling is performed on each modality according to the text feature sequence, audio feature sequence and visual feature sequence, and the semantic similarity between the discourse semantics of different nodes in the same modality is measured by cosine similarity to obtain a semantic similarity score; a similarity matrix of each modality is constructed according to the semantic similarity score between the discourse of the current node of each modality and the discourse of other nodes in the conversation; a semantic relationship graph of each modality is constructed according to the similarity matrix obtained by the corresponding semantic modeling of each modality, and the edge connections and edge relationships between different nodes in the semantic relationship graph are established according to the semantic similarity scores between the corresponding nodes in the similarity matrix; The cross-modal fusion module is used to input the discourse dependency graph, psychological dependency graph and semantic relationship graph of each modality into the multimodal feature fusion device, and perform graph convolution operations in sequence to obtain different types of predicted modal features of each modality. The formula is: ； in, For modal The following modeling types are node The predicted modal features of Modeling types include structural modeling, speaker modeling, and semantic modeling. For modalities, including audio modalities , Visual Mode and text modal ; is the Sigmoid activation function; For modal The following modeling types are node Hidden representation of ; for The weight matrix of express Belong to modal one; With Node The set of neighbor nodes of node Other nodes that belong to the same conversation; is the normalization constant, For modal The following modeling types are node The hidden representation of the neighbor nodes of for The weight matrix of The features of different types of predicted modal features belonging to the same modality are fused to obtain text modal fusion features, audio modal fusion features, and visual modal fusion features. The formula is: ； in, For Node Modal The fusion characteristics of For modalities, including audio modalities , Visual Mode and text modal ; For modal The modeling type is the structural modeling node The fusion characteristics of For modal The modeling type is speaker modeling node The fusion characteristics of For modal The modeling type is semantic modeling node The fusion characteristics of Represents a splicing operation; The text modality fusion feature is used as the query feature, the audio modality fusion feature is used as the key feature, and the visual modality fusion feature is used as the value feature, and is input into the multimodal attention layer in the multimodal feature fuser; The classification module is used to input the output features of the multimodal attention layer into the sentiment classifier to obtain the predicted sentiment label.

Images