Multi-modal emotion recognition method and system based on maximizing mutual information and contrastive learning

Abstract

The application provides a multimodal emotion recognition method and system based on maximizing mutual information and contrast learning. The method comprises the following steps: constructing a multimodal emotion recognition model and a training set; using a preset feature extraction network to extract corresponding multimodal features from the multimodal data of each object and inputting the multimodal features into the multimodal emotion recognition model; using a mutual information maximization module to maximize the mutual information between any two modal features of the same object; using a modal specific information reservation module to reserve the specific information of each modal feature itself; using a fusion network to fuse and predict the three modal features of the same object; using a contrast learning module to learn the modal feature representation according to the fusion features of different objects; designing a total loss function of the multimodal emotion recognition model; training the multimodal emotion recognition model using the training set; and obtaining an emotion recognition prediction result by using the trained emotion recognition model according to the multimodal data of an object to be recognized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal emotion recognition technology, and in particular to a multimodal emotion recognition method and system based on maximizing mutual information and contrastive learning. Background Technology

[0002] With the development of technologies such as information processing, big data, and artificial intelligence, the demand for automated analysis of users' emotions regarding products or services is increasing. Although significant progress has been made in emotion recognition using unimodal methods, a single data source may not provide enough information to fully understand human emotions. To improve the accuracy, robustness, and applicability of emotion recognition, multimodal emotion recognition has become an important research area.

[0003] Most current multimodal emotion recognition methods employ complex fusion strategies, which introduce a large number of parameters, potentially increasing computational complexity and affecting model efficiency. Furthermore, complex fusion strategies may lead to overfitting of the model to the training data, resulting in inaccurate predictions. Summary of the Invention

[0004] To improve the accuracy of emotion recognition results, this invention provides a multimodal emotion recognition method and system based on maximizing mutual information and contrastive learning.

[0005] On the one hand, the present invention provides a multimodal emotion recognition method based on maximizing mutual information and contrastive learning, comprising:

[0006] Step 1: Construct a multimodal emotion recognition model, including a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module;

[0007] Step 2: Construct a multimodal data training set, which contains multimodal data of several objects, and the multimodal data of each object includes text signals, audio signals and video signals;

[0008] Step 3: Using a preset feature extraction network, extract the corresponding text features, audio features, and video features based on the multimodal data of each object, and input them into the multimodal emotion recognition model;

[0009] Step 4: Employ a mutual information maximization module to maximize the mutual information between any two modal features of the same object, and design a loss function for the mutual information maximization module; employ a modality-specific information preservation module to preserve the specific information of each modality feature itself, and design a loss function for the modality-specific information preservation module; employ a fusion network to fuse and predict the three modal features of the same object, obtaining fused features and sentiment prediction results, and design a task loss function; employ a contrastive learning module to learn modal feature representations based on the fused features of different objects, and design a loss function for the contrastive learning module.

[0010] Step 5: Design the total loss function of the multimodal emotion recognition model. The total loss function consists of the loss function of the mutual information module, the loss function of the modality-specific information module, the task loss function, and the loss function of the contrastive learning module.

[0011] Step 6: Train the multimodal emotion recognition model using the multimodal training set to obtain a trained multimodal emotion recognition model;

[0012] Step 7: Based on the multimodal data of the object to be identified, use the trained emotion recognition model to obtain the emotion recognition prediction result.

[0013] Furthermore, multimodal data of objects are extracted from video data.

[0014] Furthermore, the preset feature extraction network includes BERT and sLSTM; wherein, BERT is used to extract text features f based on the text signal. t The sLSTM is used to extract audio features f from the audio signal and video signal respectively. a and video features f v .

[0015] Furthermore, in step 4, the maximum mutual trust module is used to maximize the mutual information between any two modal features, specifically including:

[0016] Calculate the lower bound of the mutual information between two modal features, and use the lower bound of the mutual information to approximate the mutual information between the two modal features;

[0017]

[0018] Where X and Y represent the feature vectors of two modal features, E p(x,y) [log q(y|x)]+H(Y) is the lower bound of the mutual information, denoted by I. MI q(y|x) represents the approximate distribution, H(Y) represents the differential entropy of Y; E p(x,y) Expressing expectations;

[0019] And, the loss function of the mutual information maximization module is:

[0020]

[0021] in, This represents the mutual information between text features and video features. This represents the mutual information between text features and audio features. This represents the mutual information between video features and audio features.

[0022] Furthermore, in step 4, a modality-specific information retention module is used to retain specific information about each modality feature, specifically including:

[0023] Text features f of the same object t Audio features f a and video features f v Mapping to the same dimension and then performing L2 normalization on each yields new text features. Audio features and video features

[0024] Furthermore, the loss function for retaining modality-specific information modules is:

[0025]

[0026] Where E represents the expectation of N objects, and a represents the modality gap between different modalities.

[0027] Furthermore, in step 4, a fusion network is used to fuse the three modal features of the same object, specifically including:

[0028] Text features f of the same object t Audio features f a and video features f v The features are concatenated along the column direction and then input into the fusion network for fusion and prediction.

[0029] And, the task loss function is:

[0030] L task =MAE(y′,y)

[0031] Where MAE represents the mean absolute error, y′ represents the sentiment prediction value of the fusion network, and y represents the actual sentiment value of the object.

[0032] Furthermore, in step 4, a contrastive learning module is used to learn modal feature representations based on the fusion features of different objects, specifically including:

[0033] Define anchor, positive sample, and negative sample, and perform L2 normalization on anchor, positive sample, and negative sample respectively to obtain

[0034] And, the loss function of the contrastive learning module:

[0035]

[0036] in, This represents the anchor after L2 normalization. This represents the positive samples after L2 normalization. Represents the positive and negative samples after L2 normalization, 1 [k≠i] ∈{0,1} represents an indicator function, which takes the value 1 when k≠i and takes the value 0 when k=i. Z represents the anchor, positive sample, and negative sample corresponding to each sample in each batch.

[0037] Specifically, in contrastive learning, each sample typically has one positive sample and several negative samples. The loss is defined by minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs.

[0038] Furthermore, the definition of anchor, positive sample, and negative sample specifically includes:

[0039] Text features, audio features, and video features of the same object are concatenated in the column direction and input into the fusion network. The fused features output by the fusion network are used as anchors.

[0040] Data augmentation is performed on the text, audio, and video features of the same object. The augmented text, audio, and video features are then concatenated in the column direction and input into the fusion network. The fused features output by the fusion network are used as positive samples.

[0041] Extract any two modal features from one object and the last modal feature from another object. Concatenate the three modal features in the column direction and input them into the fusion network. The fused features output by the fusion network are used as negative samples.

[0042] Furthermore, in step 5, the total loss function is:

[0043] L main =L task +αL MI +βL S +γL N

[0044] Among them, L task Indicates task loss, L MIL represents the loss of the module that maximizes mutual information. S L represents the loss of the module that retains modality-specific information. N denoted by , where α, β, and γ represent the loss of the contrastive learning module, and α, β, and γ represent the hyperparameters that control the effect of the loss function.

[0045] On the other hand, the present invention provides a multimodal emotion recognition system based on maximizing mutual information and contrastive learning, comprising:

[0046] A multimodal emotion recognition model construction module is used to build a multimodal emotion recognition model, including a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module. Specifically, the mutual information maximization module maximizes the mutual information between any two modal features of the same object, and a loss function for the mutual information maximization module is designed. The modality-specific information preservation module preserves the specific information of each modality feature, and a loss function for the modality-specific information preservation module is designed. The fusion network fuses and predicts the three modal features of the same object to obtain fused features and emotion prediction results, and a task loss function is designed. The contrastive learning module learns modal feature representations based on the fused features of different objects, and a loss function for the contrastive learning module is designed. Finally, a total loss function for the multimodal emotion recognition model is designed, which consists of the loss functions of the mutual information maximization module, the modality-specific information preservation module, the task loss function, and the contrastive learning module.

[0047] The training set construction module is used to construct a multimodal data training set, which contains multimodal data of several objects, and the multimodal data of each object includes text signals, audio signals and video signals;

[0048] The feature extraction module is used to extract corresponding text features, audio features, and video features based on the multimodal data of each object using a preset feature extraction network, and input them into the multimodal emotion recognition model;

[0049] The training and prediction module uses the multimodal training set to train the multimodal emotion recognition model to obtain a trained multimodal emotion recognition model; based on the multimodal data of the object to be recognized, the trained emotion recognition model is used to obtain the emotion recognition prediction result.

[0050] The beneficial effects of this invention are:

[0051] This invention maximizes mutual information between modalities while preserving modal-specific information to reduce the loss of valuable relevant task information. It also utilizes contrastive learning to distinguish different objects and learn modal feature representations. This emotion recognition method achieves higher accuracy in predicting emotions. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the multimodal emotion recognition method based on maximizing mutual information and contrastive learning provided in an embodiment of the present invention.

[0053] Figure 2 This is a framework diagram of the multimodal emotion recognition model provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment of the invention provides a multimodal emotion recognition method based on maximizing mutual information and contrastive learning, including the following steps:

[0057] S101: Construct a multimodal emotion recognition model, including a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module; such as Figure 2 As shown.

[0058] S102: Construct a multimodal data training set, which contains multimodal data of several objects, wherein the multimodal data of each object includes a text signal X. t Audio signal X a and video signal X v ;

[0059] Specifically, human emotions can be expressed through facial expressions, speech, or body language. Based on this, this embodiment selects to extract multimodal data of the same object from video data. This ensures that there is an inherent connection between the extracted text signals, audio signals, and video signals, thereby improving the accuracy of emotion recognition results.

[0060] S103: A preset feature extraction network is used to extract corresponding text features, audio features, and video features based on the multimodal data of each object, and these features are then input into the multimodal emotion recognition model.

[0061] Specifically, in this embodiment, the designed feature extraction network includes BERT and sLSTM; wherein, the BERT is used to extract text features f based on the text signal. tThe sLSTM is used to extract audio features f from the audio signal and video signal respectively. a and video features f v .like Figure 2 As shown.

[0062] S104: Employ a mutual information maximization module to maximize the mutual information between any two modal features of the same object, and design a loss function for the mutual information maximization module; employ a modality-specific information preservation module to preserve the specific information of each modality feature itself, and design a loss function for the modality-specific information preservation module; employ a fusion network to fuse and predict the three modal features of the same object, obtain fused features and sentiment prediction results, and design a task loss function; employ a contrastive learning module to learn modal feature representations based on the fused features of different objects, and design a loss function for the contrastive learning module;

[0063] Specifically, in general, since the extracted multimodal features are high-dimensional features, it is difficult to directly calculate the mutual information between any two modal features in a high-dimensional space. Therefore, this embodiment uses the lower bound of mutual information to approximate the calculation of mutual information.

[0064] Let X and Y represent the feature vectors of any two modal features, and maximize the mutual information between text features and video features, text features and audio features, and video features and audio features. For example, to maximize the mutual information between text features and video features, text features are considered as X and video features as Y; to maximize the mutual information between text features and audio features, text features are considered as X and audio features as Y; to maximize the mutual information between video features and audio features, video features are considered as X and audio features as Y. Correspondingly, a maximum mutual information module is used to maximize the mutual information between any two modal features, specifically including: calculating the lower bound of the mutual information between two modal features, and using the lower bound of the mutual information to approximate the mutual information between the two modal features;

[0065]

[0066] Among them, E p(x,y) [log q(y|x)]+H(Y) is the lower bound of the mutual information, denoted by I. MI p(y|x) represents the true distribution, q(y|x) represents the approximate distribution, H(Y) represents the differential entropy of Y, and H(Y|X) represents the conditional entropy; E p(x,y) This represents the expectation. It is understandable that when q(y|x) = p(y|x), the lower bound of the mutual information equals the true value.

[0067] And, the loss function of the mutual information maximization module is:

[0068]

[0069] in, This represents the mutual information between text features and video features. This represents the mutual information between text features and audio features. This represents the mutual information between video features and audio features.

[0070] As can be seen from the above, as long as "E" p(x,y) By maximizing [log q(y|x)]+H(Y), the mutual information between the two modal features can be obtained.

[0071] For mutual information lower bound I MI The first item E in p(x,y) [log q(y|x)], the calculation process is as follows: Represent q(y|x) as a multivariate Gaussian distribution. I represents the identity matrix; the mutual information maximization module is implemented using a neural network; the learnable parameters of this neural network include θ1 and θ2. By adjusting the parameter θ1, the neural network can predict the mean; by adjusting the parameter θ2, the neural network can predict the variance. This neural network contains two linear layers and one ReLU activation function layer. Thus, the first term is equivalent to the following likelihood maximization loss function:

[0072]

[0073] For mutual information lower bound I MI The second term H(Y) is calculated as follows: a Gaussian mixture model is used. Emotional polarity (positive, neutral, negative) is chosen as the classification criterion, and a Gaussian distribution N is constructed for each class. pos (μ1,∑1),N neu (μ2,∑2) and N neg (μ3,∑3).

[0074]

[0075]

[0076] Where μ and ∑ represent the mean and covariance matrix of the sentiment polarity class, respectively, c∈{pos,neu,neg} represents the sentiment polarity class to which the sample belongs, and N c This represents the number of samples in class c. This represents the feature vector of a certain mode of the i-th sample in class c.

[0077] The entropy of a multivariate Gaussian distribution is given by:

[0078]

[0079] Where k represents the dimension of the vector in the Gaussian mixture model, and det(∑) represents the determinant of ∑.

[0080] The lower bound of the entropy of the Gaussian mixture model is:

[0081]

[0082] Among them, w c h represents the probability of class c. c The entropy represents the subdistribution of class c.

[0083] Using the lower bound of the Gaussian mixture model entropy as an approximation, the final Gaussian mixture model entropy H(Y) is obtained:

[0084]

[0085] Following the above process, the mutual information between text features and video features can be calculated separately. Mutual information between text features and audio features Mutual information between video features and audio features

[0086] In this embodiment, the loss function for maximizing the mutual information module is:

[0087] The module for preserving modality-specific information specifically includes: preserving the text features f of the same object. t Audio features f a and video features f v Mapping to the same dimension and then performing L2 normalization on each yields new text features. Audio features and video features

[0088] Specifically, the dimensions of the different modal features output by the feature extraction network are likely to be inconsistent. For example, in one implementation, for a text signal, BERT is used to encode the input sentence, and the output of the last layer is used as the text feature vector, which is finally represented as a 768-dimensional feature vector. For audio signals, a unidirectional long short-term memory (sLSTM) network is used to capture the temporal features of the audio modalities, which are ultimately represented as 16-dimensional feature vectors. For video signals, a unidirectional long short-term memory (sLSTM) network is used to capture the temporal features of the video modalities, which are ultimately represented as 16-dimensional feature vectors. Considering the inconsistency of dimensions, dimension unification and normalization were performed here to facilitate the calculation of the loss of this module.

[0089] As one possible implementation, a neural network G is used to process the text modality feature vector f t Audio modal feature vector f a and video modal feature vector f v Mapping to the same dimension yields f′ t f′ a and f′ v :f′ t =G(f t f′ a =G(f a f′ v =G(f v The neural network G contains a batch normalization layer, a dropout layer, and three linear layers, each of which is activated by the ReLU function. Then, f′ is processed separately. t f′ a and f′ v Perform L2 normalization so that the dot product of any two modes is between 0 and 1, and obtain... and

[0090] In this embodiment, the loss function for retaining modality-specific information modules is:

[0091]

[0092] Where E represents the expectation of N objects, and a represents the modality gap between different modalities.

[0093] In fusion networks, this specifically includes: incorporating text features f t Audio features f a and video features f v The feature vectors are concatenated along the column direction and then input into the fusion network to predict the sentiment y'.

[0094] y′=F(f t ,f v ,f a )

[0095] Where F represents a converged network.

[0096] In this embodiment, the task loss function is:

[0097] L task =MAE(y′,y)

[0098] Where MAE represents the mean absolute error, y′ represents the sentiment prediction value of the fusion network, and y represents the actual sentiment value of the object.

[0099] The contrastive learning module specifically includes: defining anchors, positive samples, and negative samples, and performing L2 normalization on the anchors, positive samples, and negative samples respectively. z represents the anchor, positive sample, or negative sample before normalization.

[0100] Specifically, the text features f of the same object t Audio features f a and video features f v The data is concatenated along the column direction and input into the fusion network. The fused features output by the fusion network are used as anchors.

[0101] Text features f of the same object t Audio features f a and video features f v Data augmentation is performed by concatenating the augmented text features, audio features, and video features in the column direction and inputting them into the fusion network. The fused features output by the fusion network are used as positive samples.

[0102] Extract any two modal features from one object, extract the last modal feature from another object, concatenate the three modal features in the column direction, and input them into the fusion network. The fused feature output by the fusion network is used as a negative sample. For example, extract text and audio features from object i, extract video features from object j, and then concatenate the three modal features in the column direction and input them into the fusion network to obtain a negative sample.

[0103] In this embodiment, the loss function of the contrastive learning module is:

[0104]

[0105] in, This represents the anchor after L2 normalization. This represents the positive samples after L2 normalization. Represents the positive and negative samples after L2 normalization, 1 [k≠i] ∈{0,1} represents an indicator function, which takes the value 1 when k≠i and takes the value 0 when k=i. Z represents the anchor, positive sample, and negative sample corresponding to each sample in each batch.

[0106] S105: Design the total loss function of the multimodal emotion recognition model, wherein the total loss function consists of the loss function of the mutual information module, the loss function of the modality-specific information module, the task loss function, and the loss function of the contrastive learning module;

[0107] Specifically, the total loss function is:

[0108] L main =L task +αL MI +βL S +γL N

[0109] Among them, L task Indicates task loss, L MI L represents the loss of the module that maximizes mutual information. S L represents the loss of the module that retains modality-specific information. N denoted by , where α, β, and γ represent the loss of the contrastive learning module, and α, β, and γ represent the hyperparameters that control the effect of the loss function.

[0110] S106: The multimodal emotion recognition model is trained using the multimodal training set to obtain a trained multimodal emotion recognition model;

[0111] Specifically, during the training phase, the learning parameters of the multimodal emotion recognition model are continuously optimized:

[0112]

[0113] S107: Based on the multimodal data of the object to be identified, the trained emotion recognition model is used to obtain the emotion recognition prediction result.

[0114] Mutual information measures the dependency between two random variables. A higher mutual information value indicates a stronger correlation between the two random variables, meaning a higher degree of dependency. By maximizing the mutual information between any two modalities, modality-invariant content across modalities can be preserved as much as possible. Contrastive learning is used to distinguish different objects to learn modal feature representations, thereby learning each modality and the information relationships between modalities. Therefore, the emotion recognition method provided in this invention maximizes the mutual information between modalities while preserving modality-specific information to reduce the loss of valuable relevant task information, and utilizes contrastive learning to distinguish different objects to learn modal feature representations. This emotion recognition method achieves higher accuracy in predicting emotions.

[0115] Example 2

[0116] Corresponding to the above methods, this embodiment of the invention also provides a multimodal emotion recognition system based on maximizing mutual information and contrastive learning, including a multimodal emotion recognition model construction module, a training set construction module, a feature extraction module, and a training and prediction module.

[0117] The multimodal emotion recognition model construction module is used to construct a multimodal emotion recognition model, which includes a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module. Specifically, the mutual information maximization module maximizes the mutual information between any two modal features of the same object, and a loss function for the mutual information maximization module is designed. The modality-specific information preservation module preserves the specific information of each modality feature, and a loss function for the modality-specific information preservation module is designed. The fusion network fuses and predicts the three modal features of the same object to obtain fused features and emotion prediction results, and a task loss function is designed. The contrastive learning module learns modal feature representations based on the fused features of different objects, and a loss function for the contrastive learning module is designed. Finally, the total loss function of the multimodal emotion recognition model is designed, which consists of the loss functions of the mutual information maximization module, the modality-specific information preservation module, the task loss function, and the contrastive learning module.

[0118] The training set construction module is used to construct a multimodal data training set, which contains multimodal data of several objects, and the multimodal data of each object includes text signals, audio signals and video signals;

[0119] The feature extraction module is used to extract corresponding text features, audio features, and video features based on the multimodal data of each object using a preset feature extraction network, and input them into the multimodal emotion recognition model;

[0120] The training and prediction module uses the multimodal training set to train the multimodal emotion recognition model to obtain a trained multimodal emotion recognition model; based on the multimodal data of the object to be recognized, the trained emotion recognition model is used to obtain the emotion recognition prediction result.

[0121] It should be noted that the multimodal emotion recognition system based on maximizing mutual information and contrastive learning provided in this embodiment of the invention is for implementing the above method. Its specific functions can be referred to the above method embodiments, and will not be repeated here.

[0122] To verify the effectiveness of the present invention, the multimodal emotion recognition model (hereinafter referred to as MICL) provided by the present invention is compared with existing multimodal emotion recognition methods.

[0123] The following sections will elaborate on the datasets and evaluation metrics used, and analyze the advantages of the model in this invention compared to existing multimodal emotion recognition models.

[0124] Two commonly used datasets, CUM-MOSI and CMU-MOSEI, were selected: CMU-MOSI contains 2199 discourse video clips sliced ​​from 93 videos, including 89 different narrators sharing their opinions on interesting topics. Each clip is manually annotated with a sentiment value from -3 to +3, indicating the polarity and relative intensity of the sentiment. The CMU-MOSEI dataset is an upgrade of CMU-MOSI by expanding the size of the dataset; it consists of 23454 movie review video clips from YouTube.

[0125] The evaluation metrics for the CMU-MOSI and CMU-MOSEI datasets are the same. Various metrics are applied to evaluate the performance of each model: (1) Mean Absolute Error (MAE), the average absolute difference between the predicted and true values; (2) Pearson Correlation (Corr), which measures the degree of prediction bias; (3) Acc-7 accuracy, the ratio of correct predictions to the corresponding true values ​​falling into seven intervals between -3 and +3; (4) Binary Accuracy (ACC-2), the proportion of correct predictions; and (5) F1 score.

[0126] Table 1 Performance of multimodal models on the CMU-MOSI dataset

[0127]

[0128]

[0129] Table 2 Performance of multimodal models on the CMU-MOSEI dataset

[0130] TFN 0.610 0.671 49.80 79.40 79.70 LMF 0.608 0.677 50.00 80.60 81.00 QMF 0.640 0.658 47.90 80.70 79.80 MMIM 0.551 0.752 53.02 82.27 82.30 MICL 0.538 0.765 53.57 83.71 83.86

[0131] The above comparison shows that the MICL model of this invention performs well on both datasets.

[0132] Next, we will further analyze parameter a, which preserves the modality-specific information.

[0133] Table 3 Selection of modal gap α in CMU-MOSI dataset

[0134] a＝0.6 0.742 0.770 45.63 82.51 82.45 a＝0.7 0.739 0.779 46.79 82.94 82.85 a＝0.8 0.701 0.801 46.79 84.26 84.16 a＝0.9 0.73 0.784 44.17 83.09 83.1

[0135] The above comparison shows that the model achieves optimal performance when 'a' is set to 0.8. Table 3 shows that setting 'a' too high may result in the loss of modal-specific information to align different modes, while setting 'a' too low leads to significant modal discrepancies, causing a decrease in model performance.

[0136] Finally, we analyze the impact of each module on the model:

[0137] Table 4 Ablation studies using the CMU-MOSI dataset

[0138]

[0139]

[0140] The above experiments show that removing any loss from the total loss will decrease the model's performance, indicating that every module in the model is crucial.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal emotion recognition method based on maximizing mutual information and contrastive learning, characterized in that, include: Step 1: Construct a multimodal emotion recognition model, including a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module; Step 2: Construct a multimodal data training set, which contains multimodal data of several objects, and the multimodal data of each object includes text signals, audio signals and video signals; Step 3: Using a preset feature extraction network, extract the corresponding text features, audio features, and video features based on the multimodal data of each object, and input them into the multimodal emotion recognition model; Step 4: Employ a mutual information maximization module to maximize the mutual information between any two modal features of the same object, and design a loss function for the mutual information maximization module; employ a modality-specific information preservation module to preserve the specific information of each modality feature itself, and design a loss function for the modality-specific information preservation module; employ a fusion network to fuse and predict the three modal features of the same object, obtaining fused features and sentiment prediction results, and design a task loss function; employ a contrastive learning module to learn modal feature representations based on the fused features of different objects, and design a loss function for the contrastive learning module. In step 4, a modality-specific information retention module is used to retain the specific information of each modality feature, specifically including: Text features of the same object Audio features and video features Map to the same dimension, then perform separate operations. Normalization yields new text features. Audio features and video features ; Furthermore, the loss function for retaining modality-specific information modules is: in, express The expectation of an object Indicates the modal difference between different modes; In step 4, a contrastive learning module is used to learn modal feature representations based on the fusion features of different objects, specifically including: Define anchor, positive sample, and negative sample, and process anchor, positive sample, and negative sample respectively. Normalization yields ; And, the loss function of the contrastive learning module: in, Indicates the process Normalized anchor Indicates the process Normalized positive samples, Indicates the process Normalized positive and negative samples, Represents an indicator function, when When, the value is 1; when When, the value is 0. This represents the anchor, positive sample, and negative sample corresponding to each sample in each batch; The definitions of anchor, positive sample, and negative sample specifically include: Text features, audio features, and video features of the same object are concatenated in the column direction and input into the fusion network. The fused features output by the fusion network are used as anchors. Data augmentation is performed on the text, audio, and video features of the same object. The augmented text, audio, and video features are then concatenated in the column direction and input into the fusion network. The fused features output by the fusion network are used as positive samples. Extract any two modal features from one object, extract the last modal feature from another object, concatenate the three modal features in the column direction and input them into the fusion network. The fused features output by the fusion network are used as negative samples. Step 5: Design the total loss function of the multimodal emotion recognition model. The total loss function consists of the loss function of the mutual information module, the loss function of the modality-specific information module, the task loss function, and the loss function of the contrastive learning module. Step 6: Train the multimodal emotion recognition model using the multimodal training set to obtain a trained multimodal emotion recognition model; Step 7: Based on the multimodal data of the object to be identified, use the trained emotion recognition model to obtain the emotion recognition prediction result.

2. The multimodal emotion recognition method based on maximizing mutual information and contrastive learning according to claim 1, characterized in that, Extracting multimodal data of objects from video data.

3. The multimodal emotion recognition method based on maximizing mutual information and contrastive learning according to claim 1, characterized in that, The preset feature extraction network includes BERT and sLSTM; wherein, BERT is used to extract text features from the text signal. The sLSTM is used to extract audio features from the audio and video signals respectively. and video features .

4. The multimodal emotion recognition method based on maximizing mutual information and contrastive learning according to claim 1, characterized in that, In step 4, the maximum mutual trust module is used to maximize the mutual information between any two modal features, specifically including: Calculate the lower bound of the mutual information between two modal features, and use the lower bound of the mutual information to approximate the mutual information between the two modal features; in, and These represent the feature vectors of the two modal features. The lower bound of mutual information is denoted by . ; Represents an approximate distribution. express The differential entropy; Expressing expectations; And, the loss function of the mutual information maximization module is: in, This represents the mutual information between text features and video features. This represents the mutual information between text features and audio features. This represents the mutual information between video features and audio features.

5. The multimodal emotion recognition method based on maximizing mutual information and contrastive learning according to claim 1, characterized in that, In step 4, a fusion network is used to fuse the three modal features of the same object, specifically including: Text features of the same object Audio features and video features The features are concatenated along the column direction and then input into the fusion network for fusion and prediction. And, the task loss function is: Where MAE represents the mean absolute error. This represents the sentiment prediction value of the fusion network. Represents the object's true emotional value.

6. The multimodal emotion recognition method based on maximizing mutual information and contrastive learning according to claim 1, characterized in that, In step 5, the total loss function is: in, Indicates mission loss. This represents the loss of the module that maximizes mutual information. This represents the loss of the module that retains modality-specific information. This represents the loss of the contrastive learning module. , and This represents the hyperparameters that control the effect of the loss function.

7. A multimodal emotion recognition system based on maximizing mutual information and contrastive learning, characterized in that, Applied to the method of claim 1, comprising: A multimodal emotion recognition model construction module is used to build a multimodal emotion recognition model, including a mutual information maximization module, a modality-specific information preservation module, a fusion network, and a contrastive learning module. Specifically, the mutual information maximization module maximizes the mutual information between any two modal features of the same object, and a loss function for the mutual information maximization module is designed. The modality-specific information preservation module preserves the specific information of each modality feature, and a loss function for the modality-specific information preservation module is designed. The fusion network fuses and predicts the three modal features of the same object to obtain fused features and emotion prediction results, and a task loss function is designed. The contrastive learning module learns modal feature representations based on the fused features of different objects, and a loss function for the contrastive learning module is designed. Finally, a total loss function for the multimodal emotion recognition model is designed, which consists of the loss functions of the mutual information maximization module, the modality-specific information preservation module, the task loss function, and the contrastive learning module. The training set construction module is used to construct a multimodal data training set, which contains multimodal data of several objects, and the multimodal data of each object includes text signals, audio signals and video signals; The feature extraction module is used to extract corresponding text features, audio features, and video features based on the multimodal data of each object using a preset feature extraction network, and input them into the multimodal emotion recognition model; The training and prediction module uses the multimodal training set to train the multimodal emotion recognition model to obtain a trained multimodal emotion recognition model; based on the multimodal data of the object to be recognized, the trained emotion recognition model is used to obtain the emotion recognition prediction result.

Images