Multi-modal emotion recognition method based on deep learning

Abstract

The invention relates to the technical field of emotion recognition, in particular to a multi-modal emotion recognition method based on deep learning, which comprises the following steps: constructing and training a multi-modal emotion recognition model, and inputting to-be-recognized data into the trained multi-modal emotion recognition model to output a recognition result; the multi-modal emotion recognition model comprises a self-attention encoder, a first cross attention encoder, a second cross attention encoder, a gating unit and an MLP classifier; according to the method, comprehensive learning and interaction are carried out by combining two modal data of voice and text, the emotion information is more comprehensive, the inherent defect of a single feature is made up, and the robustness of the model is higher and the accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method based on deep learning. Background Art

[0002] Emotion is a high-level activity of the human brain and a complex psychological and physiological state of human beings. With the vigorous development of deep learning theory, models based on classic networks such as convolutional neural networks, recurrent neural networks, feedforward neural networks and their variants have shined in single-modal emotion recognition tasks; however, the continuous development of social networks has made people express their emotions in more and more diverse ways, and there are differences in the way different individuals express their emotions. Traditional single emotion recognition models, such as using only the time domain, frequency domain and related characteristics of speech signals, are no longer sufficient to identify complex emotional information; and, especially speech, emotion recognition methods that rely on the statistical characteristics of single-modal data rely on the development of human sound signal extraction technology. This method not only requires separate processing in stages, which is cumbersome and complex; but also does not have the characteristics related to emotion recognition tasks, which can easily become a bottleneck for subsequent model learning. For this reason, multimodal emotion recognition came into being. Multimodal emotion recognition is a way to break the data barriers between modalities and integrate multiple data features for emotion recognition. Generally, multimodal emotion recognition methods will add text information and facial expressions to speech signals to convey richer emotions and capture more information that may be contained in speech. However, some existing multimodal emotion recognition methods place too much emphasis on the fusion of different modal features and ignore the learning of rich emotional information contained in a single modality, resulting in insufficient learning of single-modal emotional information by the underlying network of the model; in the feature processing stage, the fusion is performed by splicing and weight addition, which is too simple and crude, and there is a problem of information loss. In addition, emotional features and non-emotional features (speaker, gender, etc.) are mixed together and input into the upper fusion module and classifier, making it difficult to classify the model. In addition, some multimodal emotion recognition methods use multiple classifiers to predict and score different modalities, and obtain the final emotional prediction result by voting; this solution easily causes too many model parameters and brings huge performance overhead. Summary of the invention

[0003] To solve the above problems, the present invention provides a multimodal emotion recognition method based on deep learning, constructs and trains a multimodal emotion recognition model, inputs the data to be recognized into the trained multimodal emotion recognition model to output the recognition result; the multimodal emotion recognition model includes a self-attention encoder, a first cross-attention encoder, a second attention encoder, a gating unit and an MLP classifier;

[0004] The training process of the multimodal emotion recognition model includes the following steps:

[0005] S1. Acquire modal data, wherein the modal data includes speech modal data and its corresponding text modal data; use a pre-trained model to embed features of the modal data to obtain embedded features, wherein the embedded features include speech embedding features and text embedding features;

[0006] S2. Input the embedded features into the self-attention encoder to obtain the intra-modal representation, wherein the intra-modal representation includes the intra-modal representation of the speech modality and the intra-modal representation of the text modality;

[0007] S3. Inputting the intra-modal representation and the embedded features into the first cross-attention encoder to obtain an enhanced representation, wherein the enhanced representation includes a speech enhanced representation and a text enhanced representation;

[0008] S4. Performing a dimensionality reduction operation on the enhanced representation to obtain an emotion constraint representation, wherein the emotion constraint representation includes a speech emotion constraint representation and a text emotion constraint representation; calculating the emotion constraint loss according to the emotion constraint representation;

[0009] S5. Using a second attention encoder to process the enhanced representation to obtain an interaction representation, wherein the interaction representation includes a speech interaction representation and a text interaction representation;

[0010] S6. Using a gated unit to fuse the enhanced representation and the interactive representation to obtain a fused representation, wherein the fused representation includes a speech fusion representation and a text fusion representation; splicing the speech fusion representation and the text fusion representation to obtain a fused emotion representation;

[0011] S7. Perform maximum pooling on the fused sentiment representation to obtain global features, and input the global features into the MLP classifier to obtain the score of each sentiment category; then calculate the sentiment classification loss and the supervised contrast loss;

[0012] S8. Perform weighted summation of sentiment constraint loss, sentiment classification loss and supervised contrast loss to obtain the total loss, and optimize the model parameters according to the total loss until the model parameters converge.

[0013] Beneficial effects of the present invention:

[0014] Compared with the two-stage method of feature extraction and model training currently used, the method of the present invention uses a pre-trained model to generate embedded representations of speech and text modalities for end-to-end model training, which simplifies the design complexity of the model front-end module. In addition, the method of freezing the parameters of the pre-trained model can avoid the "catastrophic forgetting" problem and reduce the training cost of the model.

[0015] Compared with the emotion prediction method based on single voice modal data, the present invention combines voice and text modal data for comprehensive learning and interaction, and the emotion information is more comprehensive, which makes up for the inherent defects of single feature, making the model more robust and more accurate.

[0016] Some existing multimodal emotion recognition methods overemphasize the integration of multimodal features and ignore the learning of rich emotional information contained in a single modality, resulting in insufficient learning of the model's underlying network for the modality and failure to fully tap the potential of the single modality. The present invention emphasizes effective intra-modal feature learning through the design of a reasonable network module structure and interaction mode, and designs an intra-modal learning module (composed of two parallel self-attention encoders and cross-attention encoders) to convert general speech and text representations into representations that are strongly related to the emotion classification task; in order to ensure that the underlying module fully learns the emotional information in speech and text, the present invention uses emotional labels as constraints to enable the module to learn representations that are strongly related to emotions, avoiding the problems of modal learning loss and semantic inconsistency. At the same time, a multimodal feature fusion module (consisting of two parallel cross-attention encoders and a gating unit) is designed to perform multimodal feature alignment and interaction to obtain complementary emotion fusion representations. Finally, the model is jointly trained by designing multiple training objectives (loss), so that the model can learn highly separable emotion representations in the feature space. Among them, the present invention adopts label smoothing technology and supervised contrastive learning for multi-objective joint training. By introducing multiple training objectives, the model is effectively constrained, the category discrimination is improved, and the generalization ability of the model is improved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a training flow chart of the method of the present invention;

[0018] Figure 2 This is a training diagram of the multimodal emotion recognition model of the present invention;

[0019] Figure 3 This is a structural diagram of the gate control unit of the present invention;

[0020] Figure 4 This is a visualization diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The present invention provides a multimodal emotion recognition method based on deep learning, including constructing and training a multimodal emotion recognition model, inputting data to be recognized into the trained multimodal emotion recognition model to output a recognition result; the multimodal emotion recognition model includes a self-attention encoder, a first cross-attention encoder, a second cross-attention encoder, a gating unit and an MLP classifier.

[0023] like Figure 1 , Figure 2 As shown, the training process of the multimodal emotion recognition model includes the following steps:

[0024] S1. Acquire modal data, wherein the modal data includes speech modal data and its corresponding text modal data; use a pre-trained model to perform feature embedding on the modal data to obtain embedded features, wherein the embedded features include speech embedding features and text embedding features.

[0025] Specifically, the pre-trained model can generate high-quality representations suitable for related downstream tasks by undergoing self-supervisory training in large-scale unlabeled data. The present invention introduces the Wav2vec2.0 pre-trained model and the Roberta pre-trained model in the field of speech and natural language processing. The Wav2vec2.0 pre-trained model is used to embed the speech modality data to obtain speech embedding features, and the Roberta pre-trained model is used to embed the text modality data to obtain text embedding features. In order to retain all information, the present invention uses 0 padding for the sequence length of speech feature embedding and text feature embedding in the training stage. Specifically, the lengths of different samples in each training batch (batch) (i.e., speech duration, text length) are different, and the embedding features of different samples obtained by the pre-training model are also different in the time series dimension. In order to perform model training, it is necessary to make the time series dimensions of all samples in the same training batch consistent; the present invention calculates and obtains the maximum length in the same training batch, and fills in 0 so that the sequence lengths of all speech embedding features and the sequence lengths of all text embedding features reach the maximum length.

[0026] The process of step S1 can be formally expressed as:

[0027]

[0028] in, represents text embedding features, X t Represents text modal data, Represents speech embedding features. X a represents speech modality data, RoBERTa(·) represents the Roberta pre-trained model, and Wav2Vec2(·) represents the Wav2vec2.0 pre-trained model.

[0029] S2. Input the embedded features into the self-attention encoder to obtain the intra-modal representation, wherein the intra-modal representation includes the intra-modal representation of speech and the intra-modal representation of text.

[0030] Specifically, in order to capture the rich emotional semantic information within the modality, the present invention designs an intra-modal learning module that focuses on the learning of emotional semantic information within the modality. The intra-modal learning module includes two parts: a self-attention encoder and a first cross-attention encoder. First, in the self-attention encoder part, the present invention uses a Transformer encoder structure to perform intra-modal sequence interaction on the input embedded features. The self-attention encoder includes a multi-head attention layer and a feedforward layer, and the multi-head attention layer and the feedforward layer are both connected with Layer Normalization. At the same time, the multi-head attention layer and the feedforward layer are respectively connected with two Layer Normalization residuals. The speech embedding features and the text embedding features are distributed and input into the self-attention encoder to obtain the speech modality intra-representation and the text modality intra-representation, which are expressed as:

[0031]

[0032] Among them, Self_Attention_Encoder(·) represents the self-attention encoder, represents the intramodal representation of speech, Represents the textual intra-modal representation.

[0033] S3. Input the intra-modal representation and the embedded features into the first cross-attention encoder to obtain an enhanced representation, wherein the enhanced representation includes a speech enhanced representation and a text enhanced representation.

[0034] Specifically, the present invention further strengthens the intra-modal interaction through a first cross-attention encoder, wherein the first cross-attention encoder includes a multi-head cross-attention layer and a feedforward layer, and a Layer Normalization is provided after the multi-head cross-attention layer and the feedforward layer, and the multi-head cross-attention layer and the feedforward layer are respectively connected with two LayerNormalization residuals; wherein in the multi-head cross-attention layer, the intra-modal representation is used as Queries, and the embedded features are used as Keys and Values, and learning is performed through the multi-head cross-attention mechanism; step S3 specifically includes:

[0035] The speech modality representation and speech embedding features are input into the first cross-attention encoder to obtain the speech enhanced representation, and the text modality representation and text embedding features are input into the first cross-attention encoder to obtain the text enhanced representation; which can be expressed as:

[0036]

[0037] Among them, Cross_Attention_Encoder(·) represents the first cross attention encoder, represents the speech enhancement representation, Represents text-enhanced representation.

[0038] S4. Perform a dimensionality reduction operation on the enhanced representation to obtain a sentiment constraint representation, wherein the sentiment constraint representation includes a speech sentiment constraint representation and a text sentiment constraint representation; and calculate the sentiment constraint loss based on the sentiment constraint representation.

[0039] Specifically, the present invention applies emotion tags to minimize the speech enhancement representation and text enhancement representation, so as to help the intra-modality learning module learn emotion-specific representations from the modality common features, and provide a stable emotion representation for the upper network to improve the performance of the model. Step S4 specifically includes:

[0040] S41. Reduce the dimension of the speech enhancement representation through two fully connected layers with ReLU activation function to obtain the speech emotion constraint representation It is expressed as:

[0041]

[0042] in, represents speech enhancement representation, W1 and W2 represent model learnable parameters, b1 and b2 represent bias units; Relu(·) represents ReLU activation function;

[0043] S42. Reduce the dimension of the text enhancement representation through two fully connected layers with ReLU activation function to obtain the text sentiment constraint representation It is expressed as:

[0044]

[0045] in, Representing text-enhanced representation;

[0046] S43. Calculate speech emotion constraint loss and text sentiment constraint loss It is expressed as:

[0047]

[0048] Among them, y represents the sentiment label, Represents the classification loss function. The present invention uses the cross entropy loss function. Other similar classification loss functions are within the scope of the same method proposed in the present invention.

[0049] S5. Use a second cross-attention encoder to process the enhanced representation to obtain an interaction representation, wherein the interaction representation includes a speech interaction representation and a text interaction representation.

[0050] Specifically, in order to achieve deep fusion of speech and text, the present invention proposes a modal interaction module based on cross-attention and gated fusion mechanism. The modal interaction module achieves feature-level deep fusion by interactively learning the enhanced representations of each modality, providing richer and more powerful emotional representations for the sentiment classification task. The modal interaction module includes a second cross-attention encoder and a gating unit, wherein step S5 uses the second cross-attention encoder to process the enhanced representation to obtain the interactive representation, which specifically includes:

[0051] S51. Take the speech enhancement representation as Queries, and the text enhancement representation as Keys and Values, and learn through the multi-head cross attention mechanism to obtain the speech interaction representation, which is expressed as:

[0052]

[0053] Where K_Cross_Attention_Encoder(·) represents the second cross attention encoder, represents the speech enhancement representation, represents text enhanced representation, represents speech interaction representation;

[0054] S52. Take the text-enhanced representation as Queries and the speech-enhanced representation as Keys and Values, and learn through the multi-head cross-attention mechanism to obtain the text interaction representation, which is expressed as:

[0055]

[0056] in, Represents text interaction representation.

[0057] S6. Use a gated unit to fuse the enhanced representation and the interactive representation to obtain a fused representation, wherein the fused representation includes a speech fusion representation and a text fusion representation; and concatenate the speech fusion representation and the text fusion representation to obtain a fused emotional representation.

[0058] Specifically, in order to fully preserve the emotional information within and between modalities, the present invention proposes a gated fusion mechanism to dynamically fuse the enhanced representation within the modality and the cross-modal representation, such as Figure 3 As shown, specifically including:

[0059] S61. Enhanced representation of speech and speech interaction representation Perform dimensionality reduction operations respectively to obtain the first speech representation and the second phonetic representation It is expressed as:

[0060]

[0061] Among them, W3 and W4 represent the linear layer weight matrix, which is used to convert and The two-dimensional feature representation is transformed into one-dimensional space; tanh(·) represents the tanh activation function;

[0062] S62. Enhance the representation of speech and speech interaction representation After concatenation, it passes through a linear layer and is activated using a sigmoid function to obtain the gating coefficient z a , using z a and (1-z a ) for the first speech representation and the second phonetic representation Multiply by position to get the speech fusion representation h a , expressed as:

[0063]

[0064] S63. Enhanced representation of text Interactive representation with text Perform dimensionality reduction operations respectively to obtain the first text representation and the second text representation It is expressed as:

[0065]

[0066] S64. Enhance the representation of text Interactive representation with text After concatenation, it passes through a linear layer and is activated using a sigmoid function to obtain the gating coefficient z t , using z t and (1-z t ) for the first text representation and the second text representation Multiply by position to get the text fusion representation h t , expressed as:

[0067]

[0068] S65. Fusion of speech representation h a and text fusion representation h t The fused sentiment representation h is obtained by concatenation:

[0069] h = concat(h a,h t )

[0070] S7. Perform maximum pooling on the fused sentiment representation to obtain global features, and input the global features into the MLP classifier to obtain the score of each sentiment category; then calculate the sentiment classification loss and supervised contrast loss.

[0071] Specifically, in order to capture global information, the present invention uses global maximum pooling (GMP) to perform pooling operations on the fused sentiment representation h, retaining the most representative part of each feature dimension in the fused sentiment representation h, so that the model focuses on the most important part of the sentiment feature; at the same time, the present invention designs a multi-layer linear layer (MLP) as a classifier, gradually scaling the feature dimension to the category number dimension, and using SoftMax to calculate the prediction score of each sentiment category. The overall formula is as follows:

[0072] u=Classifier(GMP(h))

[0073] p = Softmax(u)

[0074] Among them, u represents the global feature after the classifier, Classifier(·) represents a linear classifier with parameters, which is used to scale the feature dimension to the number of sentiment categories; GMP(·) represents the global maximum pooling calculation, Softmax(·) represents the normalized exponential function, and p represents the sentiment probability distribution vector.

[0075] Specifically, the present invention adopts a multi-task joint training method to train the model, wherein the multi-task includes a sentiment classification task and a supervised contrastive learning task, wherein:

[0076] For sentiment classification tasks, the present invention smoothes the one-hot encoded sentiment labels by introducing label smoothing technology to solve the overfitting problem and category imbalance problem in model training. Specifically, a category smoothing factor α is introduced in the cross-entropy loss. By introducing a certain degree of noise, the sensitivity of the model to a single deterministic prediction is reduced, and the generalization ability of the model to unseen samples is improved, thereby reducing the risk of overfitting. The sentiment classification loss formula after the introduction of Label Smoothing is expressed as:

[0077]

[0078] Among them, C represents the number of sentiment categories, y j represents the value corresponding to the jth position in the sentiment label vector after smoothing, p jrepresents the value of the jth position in the emotion probability distribution vector p, α represents the category smoothing factor; target represents the true label corresponding to the sample, j=target represents the jth position in the emotion label vector after smoothing is the true label corresponding to the sample;

[0079] For the supervised contrastive learning task, the present invention introduces a supervised contrastive learning method, which groups samples of the same emotion category into positive pairs (anchor-positive pairs) in the same batch during training, and groups samples of different emotion categories into negative pairs. Through the contrastive loss function, the model is encouraged to bring the representations of similar samples closer together and separate the representations of dissimilar samples, so that the model can learn highly separable emotion category vectors. Here, u i It represents the global representation after pooling the multimodal representation obtained by the feature fusion stage of the i-th sample in a batch (a batch in the model training process, consisting of multiple training samples), where the positive sample set of the i-th sample is formally represented as y i represents the sentiment label of the i-th sample, so the supervised contrast loss in a batch can be expressed as:

[0080]

[0081] Wherein, τ represents the temperature coefficient of supervised contrastive learning, which is used to adjust the stability of convergence; sim(·) represents similarity calculation, and the present invention adopts dot-product method for calculation; n represents the number of samples in the training batch.

[0082] S8. Perform weighted summation of sentiment constraint loss, sentiment classification loss and supervised contrast loss to obtain the total loss, and optimize the model parameters according to the total loss until the model parameters converge.

[0083] Specifically, the total loss formula is expressed as:

[0084]

[0085] Among them, α, β, and γ are the weight hyperparameters of the loss function, and their values ​​are in the range of [0,1].

[0086] Through the above constraints, the model method proposed in the present invention can gradually extract the emotion-related parts from the speech and text, perform deep fusion of speech and text, improve the discrimination of emotion categories, and finally realize effective and efficient emotion recognition.

[0087] In one embodiment, the present invention verifies that the method is superior to the currently popular multimodal fusion method by conducting extensive comparative experiments and ablation analysis on two publicly available emotion datasets (IEMOCAP and MELD). The IEMOCAP dataset is divided into four categories of emotions: Neu (neutral emotion), Hap&Exc (happy emotion), Sad (sad emotion), and Ang (angry emotion); while the MELD dataset currently has seven categories of emotion divisions and five categories of emotion divisions. Among the five categories, the emotions of fear (Fea) and disgust (Dis) with fewer samples are discarded; as shown in Table 1 and Table 2:

[0088] Table 1 Distribution of emotion categories in the IEMOCAP dataset

[0089]

[0090]

[0091] Table 2 Distribution of emotion categories in the MELD dataset

[0092]

[0093] On the IEMOCAP emotion dataset, this method is experimentally compared with similar models or methods in the past two years. The results are shown in Table 3:

[0094] Table 3 Performance comparison of SCMI-NET and SOTAMODELS on IEMOCAP

[0095]

[0096] In Table 3, [1] refers to the method proposed in the paper Robinnet: A Multi-modal Speech Emotion Recognition System With Speaker Recognition for Social Interactions, [2] refers to the method proposed in the paper Key-Sparse Transformer for Multimodal Speech Emotion Recognition, [3] refers to the method proposed in the paper Multi-modal speech emotion recognition using self-attention mechanism and multi-scale fusion framework, [4] refers to the method proposed in the paper Exploring Complementary Features in Multi-Modal Speech Emotion Recognition, [5] refers to the method proposed in the paper Mser: Multimodal Speech Emotion Recognition using Cross-Attention with Deep Fusion, [6] refers to the method proposed in the paper Semantic Alignment Network for Multi-Modal Emotion Recognition, and [7] refers to the method proposed in the paper Semantic-wise guidance for efficient multimodal emotion recognition with missing modalities. From the results in the table, it can be seen that on the IEMOCAP emotion dataset, this method has higher accuracy than the existing methods.

[0097] On the MELD sentiment dataset, this method is experimentally compared with similar models or methods in the past two years. The results are shown in Table 4:

[0098] Table 4 Performance comparison of SCMI-NET and SOTA MODELS on MELD

[0099]

[0100] In Table 4, [8] refers to the method proposed in the paper Graphcfc: A directed graph based cross-modal feature complementation approach for multimodal conversational emotion recognition, [9] refers to the method proposed in the paper Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations,

[10] refers to the method proposed in the paper Ctnet: Conversational transformer network for emotion recognition,

[11] refers to the method proposed in the paper Quantum-inspired neural network for conversational emotion recognition,

[12] refers to the method proposed in the paper Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis, and

[13] refers to the method proposed in the paper Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation. From the results in the table, it can be seen that on the MELD emotion dataset, the accuracy of this method is higher than that of the existing methods.

[0101] Among them, the English abbreviations UA mentioned in the table represent unweighted accuracy, WA represents weighted accuracy, WF1 represents weighted F1 score, Modal represents the input modality of the method (A represents audio, T represents text, and V represents video), Classes represents the number of sentiment classifications, Avg represents the average value of the experimental results, and SCMI-Net represents the method proposed in the present invention.

[0102] The experimental results of this method on the IEMOCAP and MELD emotion datasets are as follows:

[0103] Table 5:

[0104]

[0105] The ablation experiment is based on the overall method proposed in this invention, and the modules are disassembled and the control variable method is analyzed to verify the effectiveness of the proposed modules or methods. Among them, V1: Audio only retains the self-attention encoder and the first cross-attention encoder, which is a single-modal audio architecture for speech; V2: Text also only retains the self-attention encoder and the first cross-attention encoder, which is a single-modal text architecture for text; V3: Concat is a multi-modal representation architecture that retains the intra-modal learning module and simply splices speech and text; V4: Baseline is based on this invention, and removes the emotional constraints, emotional classification loss and supervised comparative learning method, and obtains the benchmark effect of the model; V5: The supervised contrast learning method is removed based on the present invention; V6: The present invention removes two emotional constraints; V6-1: The text sentiment constraint is removed based on the present invention; V6-2: The speech emotion constraint is removed on the basis of the present invention.

[0106] After the model proposed by the method of the present invention completes the training of the training set (model convergence), all samples in the test set are used as input, and the sentiment features of the model before the MLP classifier are output. The dimension is reduced by t-sne, and the vector is transformed from (1536,) to (2,). Then, it is visualized in a two-dimensional plane to present the classification and clustering ability of the model and verify the performance of the model.

[0107] The visualization results are as follows Figure 4 It shows that in the unimodal architecture (Audio and Text), different samples only show a slight clustering tendency, and it is difficult to form a clear classification boundary. In contrast, the baseline method (Baseline) uses a multi-head cross-attention encoder for feature alignment and a gated fusion unit to deeply fuse different feature representations within the modality, effectively integrating the emotional information from the two modalities. In the feature space, samples of the same emotional category are clustered together to form a clear classification boundary. In addition, based on the baseline method, the method of the present invention (SCMI-Net) enhances unimodal learning by adding emotional constraints, and combines supervised contrastive learning for multi-objective training, thereby improving the ability to distinguish emotional categories. This enhancement results in a more consistent emotional representation between different categories in the feature space, obtains clearer category boundaries, and improves the classification accuracy of the model.

[0108] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0109] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal emotion recognition method based on deep learning, characterized in that: Constructing and training a multimodal emotion recognition model, inputting the data to be recognized into the trained multimodal emotion recognition model to output a recognition result; the multimodal emotion recognition model includes a self-attention encoder, a first cross-attention encoder, a second cross-attention encoder, a gating unit and an MLP classifier; The training process of the multimodal emotion recognition model includes the following steps: S1. Acquire modal data, wherein the modal data includes speech modal data and its corresponding text modal data; use a pre-trained model to embed features of the modal data to obtain embedded features, wherein the embedded features include speech embedding features and text embedding features; S2. Input the embedded features into the self-attention encoder to obtain the intra-modal representation, wherein the intra-modal representation includes the intra-modal representation of the speech modality and the intra-modal representation of the text modality; S3. Inputting the intra-modal representation and the embedded features into the first cross-attention encoder to obtain an enhanced representation, wherein the enhanced representation includes a speech enhanced representation and a text enhanced representation; S4. Performing a dimensionality reduction operation on the enhanced representation to obtain an emotion constraint representation, wherein the emotion constraint representation includes a speech emotion constraint representation and a text emotion constraint representation; calculating the emotion constraint loss according to the emotion constraint representation; S5. Using a second cross-attention encoder to process the enhanced representation to obtain an interaction representation, wherein the interaction representation includes a speech interaction representation and a text interaction representation; S6. Using a gated unit to fuse the enhanced representation and the interactive representation to obtain a fused representation, wherein the fused representation includes a speech fusion representation and a text fusion representation; splicing the speech fusion representation and the text fusion representation to obtain a fused emotion representation; S7. Perform maximum pooling on the fused sentiment representation to obtain global features, and input the global features into the MLP classifier to obtain the score of each sentiment category; then calculate the sentiment classification loss and the supervised contrast loss; S8. Perform weighted summation of sentiment constraint loss, sentiment classification loss and supervised contrast loss to obtain the total loss, and optimize the model parameters according to the total loss until the model parameters converge.

2. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: Step S1 specifically includes: The Wav2vec2.0 pre-trained model is used to embed the speech modality data to obtain speech embedding features, and the Roberta pre-trained model is used to embed the text modality data to obtain text embedding features.

3. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: The self-attention encoder adopts a Transformer encoder; the speech embedding features and the text embedding features are distributed and input into the self-attention encoder to obtain the speech modality internal representation and the text modality internal representation.

4. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: The first cross-attention encoder includes a multi-head cross-attention layer and a feed-forward layer, wherein the multi-head cross-attention layer uses the intra-modal representation as Queries and the embedded features as Keys and Values, and learns through the multi-head cross-attention mechanism; step S3 specifically includes: The speech modality representation and speech embedding features are input into the first cross-attention encoder to obtain speech enhanced representation, and the text modality representation and text embedding features are input into the first cross-attention encoder to obtain text enhanced representation.

5. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: Step S4 specifically includes: S41. Reduce the dimension of the speech enhancement representation through two fully connected layers with ReLU activation function to obtain the speech emotion constraint representation It is expressed as: in, represents speech enhancement representation, W1 and W2 represent model learnable parameters, b1 and b2 represent bias units; Relu(·) represents ReLU activation function; S42. Reduce the dimension of the text enhancement representation through two fully connected layers with ReLU activation function to obtain the text sentiment constraint representation It is expressed as: in, Representing text-enhanced representation; S43. Calculate speech emotion constraint loss and text sentiment constraint loss It is expressed as: Among them, y represents the sentiment label, represents the classification loss function.

6. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: Step S5 uses the second cross-attention encoder to process the enhanced representation to obtain the interactive representation, which specifically includes: S51. Take the speech enhancement representation as Queries, and the text enhancement representation as Keys and Values, and learn through the multi-head cross attention mechanism to obtain the speech interaction representation; S52. Take the text-enhanced representation as Queries and the speech-enhanced representation as Keys and Values, and learn through a multi-head cross-attention mechanism to obtain the text interaction representation.

7. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: Step S6 specifically includes: S61. Enhanced representation of speech and speech interaction representation Perform dimensionality reduction operations respectively to obtain the first speech representation and the second phonetic representation It is expressed as: Where W3 and W4 represent the linear layer weight matrices, and tanh(·) represents the tanh activation function; S62. Enhance the representation of speech and speech interaction representation After concatenation, it passes through a linear layer and is activated using a sigmoid function to obtain the gating coefficient z a , using z a and (1-z a ) for the first speech representation and the second phonetic representation Multiply by position to get the speech fusion representation h a , expressed as: S63. Enhanced representation of text Interactive representation with text Perform dimensionality reduction operations respectively to obtain the first text representation and the second text representation It is expressed as: S64. Enhance the representation of text Interactive representation with text After concatenation, it passes through a linear layer and is activated using a sigmoid function to obtain the gating coefficient z t , using z t and (1-z t ) for the first text representation and the second text representation Multiply by position to get the text fusion representation h t , expressed as: S65. Fusion of speech representation h a and text fusion representation h t The fused sentiment representation h is obtained by concatenation.

8. The multimodal emotion recognition method based on deep learning according to claim 1, characterized in that: Sentiment classification loss The calculation formula is: Among them, C represents the number of sentiment categories, y j represents the value corresponding to the jth position in the sentiment label vector after smoothing, p j represents the value of the jth position in the sentiment probability distribution vector p, α represents the category smoothing factor, and target represents the true label corresponding to the sample; Supervised Contrastive Loss The calculation formula is: Among them, u i represents the global feature corresponding to the i-th sample in a training batch, τ represents the temperature coefficient, represents the set of positive samples of the ith sample, sim(·) represents the similarity calculation, and n represents the number of samples in the training batch.

Images