Discriminative learning based multi-modal emotion recognition method and system

Abstract

The application provides a multi-modal emotion recognition method and system based on discriminative learning. The method comprises the following steps: step 1: collecting multi-modal information, including electroencephalogram signals, facial signals, speech signals and text signals; step 2: inputting the multi-modal information into a feature extraction network respectively to obtain electroencephalogram features, facial features, speech features and text features; step 3: using a canonical correlation analysis method to calculate the correlation between any two modal features; step 4: inputting the modal features into respective corresponding single-modal classifiers respectively to obtain single-modal prediction results; step 5: using the correlation between any two modal features and the single-modal prediction results to design a class loss function corresponding to each modal; step 6: obtaining a target function according to the class loss function corresponding to each modal to guide the training of an emotion recognition model; and step 7: obtaining an emotion recognition result by using the trained emotion recognition model according to multi-modal information of an object to be recognized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal emotion recognition technology, and particularly relates to a multi-modal emotion recognition method and system based on discriminative learning. BACKGROUND

[0002] Emotion recognition methods can be divided into methods based on external behavioral information (facial expressions, speech, text) and methods based on internal physiological signals (respiratory rate, electrocardiogram, electroencephalogram). Generally, external behavioral information is more convenient and faster in data collection, but is easily controlled and intervened by humans, while physiological signals can more objectively and reliably capture the emotional state. Among physiological signals, electroencephalogram is a direct product of brain activity, containing rich spatial and temporal information, and different thinking states and emotional changes will reflect different electroencephalogram signals at different brain cortex positions. Therefore, electroencephalogram contains rich useful information and can effectively reflect the emotional state of individuals, showing outstanding performance in emotion recognition.

[0003] Current emotion recognition methods mostly use external behavioral information to distinguish emotions, but external behavioral information is easily controlled and intervened by humans and may not accurately reflect the true emotional state of the subject. In addition, in existing multi-modal emotion recognition methods, the strength of the correlation between modalities is often ignored. SUMMARY

[0004] In view of the problem that traditional emotion recognition methods have low emotion prediction accuracy due to ignoring the correlation between different modalities of information, the present application provides a multi-modal emotion recognition method and system based on discriminative learning.

[0005] In one aspect, the present application provides a multi-modal emotion recognition method based on discriminative learning, comprising:

[0006] Step 1: Collecting multi-modal information of a plurality of training subjects, the multi-modal information comprising electroencephalogram signals, facial signals, speech signals and text signals;

[0007] Step 2: Inputting the multi-modal information into respective feature extraction networks to obtain electroencephalogram features, facial features, speech features and text features;

[0008] Step 3: Using canonical correlation analysis method to calculate the correlation between any two modal features;

[0009] Step 4: Inputting each modal feature into a respective corresponding single-modality classifier to obtain a single-modality prediction result;

[0010] Step 5: Using the correlation between any two modal features and each single-modality prediction result to design a class loss function corresponding to each modality;

[0011] Step 6: Obtain a target function according to the class loss function corresponding to each modality to guide the training of the emotion recognition model.

[0012] Step 7: Obtain an emotion recognition result by using the trained emotion recognition model according to the multi-modal information of the object to be recognized.

[0013] Further, step 1 specifically comprises:

[0014] The electroencephalogram cap and the camera are used to synchronously collect the electroencephalogram and the video of the training object respectively; the face signal and the speech signal of the training object are extracted from the video, and the speech signal is converted into a text signal.

[0015] Further, in step 2, the electroencephalogram features are extracted from the electroencephalogram signal by using wavelet transform, the face features are extracted by using the OpenFace tool, the speech features are extracted by using the LibROSA tool, and the text features are extracted by using the pre-trained BERT-base model.

[0016] Further, step 2 further comprises: inputting the electroencephalogram features, the face features, the speech features and the text features into the LSTM network to learn the time dependence in each modality feature.

[0017] Further, step 3 specifically comprises:

[0018] X and Y represent the feature vectors of any two modality features, and the relationship between the two feature vectors X and Y and the corresponding to-be-solved projection vectors u and v is constructed according to formula (3):

[0019]

[0020] wherein, C XY cov(X, Y) represents the covariance of X and Y, C XX cov(X, X) represents the autocovariance of X, and C YY cov(Y, Y) represents the autocovariance of Y.

[0021] Solving formula (3) obtains u and v, and linearly combining variables X and Y by using u and v respectively obtains the reduced variables X' and Y';

[0022] The correlation degree r between any two modality features is calculated according to formula (7):

[0023]

[0024] The value range of the correlation degree is [-1, 1], and the values of-1 and 1 are strong correlation, and the value of 0 is no correlation.

[0025] Further, step 5 specifically comprises: the class loss function corresponding to each modality is as shown in formula (10):

[0026]

[0027] Wherein, is the prediction result of the y-th class according to the m-th modality feature; M is the total number of types of modality features; r t m is the correlation between the m-th modality feature and the t-th modality feature; c is a selection variable used by the correlation; β is a hyperparameter.

[0028] Further, step 6 specifically comprises: the objective function is as shown in formula (15):

[0029]

[0030] Wherein, λ m Indicates the weight, α is the penalty factor, ||·||2 represents L2 regularization, and W represents the learnable parameters in the emotion recognition model.

[0031] Further, in step 6, in the training stage, the learnable parameters are optimized according to formula (16):

[0032]

[0033] On the other hand, the present application provides a multi-modal emotion recognition system based on discriminative learning, comprising:

[0034] The acquisition module is used for acquiring multi-modal information of a plurality of training objects, and the multi-modal information includes electroencephalogram signals, face signals, voice signals and text signals.

[0035] The feature extraction module is used for inputting the multi-modal information into respective feature extraction networks to obtain electroencephalogram features, face features, voice features and text features.

[0036] The correlation calculation module is used for calculating the correlation between any two modality features using a canonical correlation analysis method.

[0037] The classifier module is used for inputting the modality features into respective corresponding single-modality classifiers to obtain single-modality prediction results.

[0038] The training and prediction module is used for designing a class loss function corresponding to each modality by using the correlation between any two modal features and the respective single-modality prediction result; a target function is obtained according to the class loss function corresponding to each modality to guide the training of the emotion recognition model; and the emotion recognition result is obtained by using the trained emotion recognition model according to the multi-modal information of the object to be recognized.

[0039] The present application has the following advantages:

[0040] The present application is based on the physiological characteristics of the electroencephalogram signal and combines external behavior information (facial expression, speech and text features) to comprehensively judge the emotion of the object, thereby avoiding the evaluation difference caused by subjective consciousness and environmental interference and other factors, and is an objective and effective emotion recognition method. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of the multi-modal emotion recognition method based on discriminative learning provided by the embodiment of the present application is provided.

[0042] Figure 2 A framework diagram of the multi-modal emotion recognition system based on discriminative learning provided by the embodiment of the present application is provided.

[0043] Figure 3 A β super parameter cross-validation selection schematic diagram of the CDL method provided by the embodiment of the present application is provided.

[0044] Figure 4 A β super parameter cross-validation selection schematic diagram of the CCDL method provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0046] Embodiment 1

[0047] As shown in the following formula (1), the present application provides a multi-modal emotion recognition method based on discriminative learning. Figure 1 The present application provides a multi-modal emotion recognition method based on discriminative learning, comprising the following steps:

[0048] S101: Collect multimodal information of several training subjects, including EEG signals, facial signals, speech signals and text signals;

[0049] Specifically, the training subjects' electroencephalogram (EEG) signals and videos are collected simultaneously using an electrode cap and a camera; facial and speech signals of the training subjects are extracted from the videos, and then the speech signals are converted into text signals.

[0050] In practical applications, EEG signals are acquired according to the international 10-20 standard system. Downsampling is performed using the eeglab toolbox to 200Hz. To filter out noise and remove artifacts, a 0-75Hz bandpass filter is used to process the EEG data, resulting in preprocessed EEG signals. Frames are extracted from the video at 30Hz, and the MTCNN face detection algorithm is used to extract aligned face signals. The MoviePy library is used to extract speech signals from the video and simultaneously convert them to text signals, such as... Figure 2 The left half is shown.

[0051] S102: Input the multimodal information into their respective feature extraction networks to obtain EEG features F. e Facial features F v Speech features F a and text features F t ;

[0052] Specifically, such as Figure 2 As shown in the feature extraction section, for EEG signals, wavelet transform is used to extract the time-frequency domain features of the EEG signals. Specifically, the EEG signal is divided into a sequence of multiple continuous segments with a duration of 30 seconds. Wavelet transform is used to calculate the wavelet time-frequency map of each 30-second segment, starting from the beginning of the original EEG signal. The wavelet time-frequency map is then input into a two-dimensional convolutional network constructed based on VGGNet, and the features output from the last layer of the network are flattened to obtain a 432-dimensional time-frequency feature vector f. e .

[0053] For face signals, the MultiComp OpenFace 2.0 toolkit was used to extract 68 facial landmarks, 17 facial motion units, head pose, head orientation, and eye gaze, resulting in a total of 709 dimensions of frame-level facial visual features. v .

[0054] For speech signals, the LibROSA speech toolkit with default parameters was used to extract acoustic features at 22050Hz, resulting in the extraction of 33-dimensional frame-level acoustic features f. a, including 1-dimensional log fundamental frequency (logF0), 20-dimensional mel-frequency cepstral coefficients (MFCCs), and 12-dimensional constant Q chromagram (CQT). These features are related to emotion and prosody.

[0055] For text signals, word vectors are obtained from the transcript using a pre-trained BERT-base model (in this embodiment, taking Chinese as an example). Due to the character-based nature of bert, Chinese word segmentation is not required, and no word segmentation tool is used here. Finally, each word is represented as a 768-dimensional word vector feature f t .

[0056] The obtained feature set Z = {F e ,F v ,F a ,F t} is input into an LSTM network composed of four long short-term memory network (LSTM) modules connected in series. For each modality feature, the LSTM encodes the modality information over time, thereby learning the temporal dependencies within the modality feature. Specifically, at each input time period t, the input m-th modality feature is The output of the LSTM network is where denotes the dimension of the LSTM module corresponding to the m-th modality feature. The LSTM network allows different modalities to have different input, storage, and output shapes.

[0057] S103: Calculate the correlation between any two modality features using the canonical correlation analysis (CCA) method:

[0058] Specifically, let X and Y represent the feature vectors of any two modality features. In this embodiment, it is assumed that X = (x1, x2, …, x p ) and Y = (y1, y2, …, y q ), and it is assumed without loss of generality that p ≤ q. CCA finds a set of projection vectors u and v to linearly combine variables X and Y, respectively, to project variables X and Y into low-dimensional isomorphic spaces, thereby obtaining new combined variables, as shown in equation (1):

[0059] X′ = u1x1 + u2x2 + … + u p x p = u T X(1)

[0060] Y′ = v1y1 + v2y2 + … + v q y q = v T Y

[0061] In formula (1), X' and Y' are canonical variables, u and v are canonical correlation coefficients, and CCA maximizes the correlation between the canonical variables X' and Y', i.e., maximizes the correlation coefficient between them, as shown in formula (2):

[0062]

[0063] After transformation, we have:

[0064]

[0065] Also expressed as:

[0066]

[0067] In formula (3), C XY = cov (X, Y) represents the covariance of X and Y, C XX = cov (X, X) represents the autocovariance of X, and C YY = cov (Y, Y) represents the autocovariance of Y. Since the ρ in formula (3) is independent of the scale of the canonical correlation coefficients u and v, formula (3) can be simplified to formula (4):

[0068]

[0069] A Lagrange function is constructed by the Lagrange multiplier method, as shown in formula (5):

[0070]

[0071] Let

[0072]

[0073]

[0074] By combining formula (4) and formula (5), under the conditions of u T C XX u = 1, v T C YY v = 1, we can simplify to λ1 = λ2, and thus obtain:

[0075]

[0076] In this way, the solution of the constrained optimization problem can be transformed into the problem of finding eigenvalues, and then the canonical correlation coefficients u and v are obtained. By linearly combining the variables using u and v, the reduced variables X' and Y' are obtained, and the correlation degree is calculated:

[0077]

[0078] The correlation value ranges from -1 to 1, and the values of -1 and 1 represent strong correlation, and the value of 0 represents no correlation. To ignore the influence of direction, the absolute value is taken, and the value range is limited to [0, 1].

[0079] S104: input each modal feature into the corresponding single-modal classifier to obtain a single-modal prediction result;

[0080] Specifically, when extracting features of each modal information, a suitable feature extraction method is selected to achieve better prediction results. Similarly, in this step, the single-modal classifier H m According to the needs of modal features, a variety of conventional classifiers can be freely selected to meet the requirements of different tasks. In this embodiment, considering the multi-classification problem to be solved and the use of information quantity in the subsequent class loss function, a softmax classifier is selected for classification processing, such as Figure 2 as shown in the right half of the figure.

[0081] A neural network with a softmax output layer is trained in advance, and the extracted modal features are input into the network for classification. The neural network has L layers, and the variable z of the last layer is denoted as:

[0082] z=w L a L-1 +b L (8)

[0083] where a L-1 is the output of the L-1 layer, w L is the weight value of the L layer, and b L is the bias value of the L layer. After calculating z, the softmax function is defined as follows:

[0084]

[0085] where z i is the output value of the i-th node, and N is the number of output nodes, i.e. the number of classes. The softmax function converts the output value of multi-classification into a probability distribution ranging from 0 to 1 and summing to 1. The result output by the softmax output layer is the single-modal prediction result.

[0086] Step 5, using the correlation between any two modal features and each single-modal prediction result, design a class loss function corresponding to each modal;

[0087] Specifically, an influence factor is constructed by using the correlation between modal features and the prediction results of each single modal, so that the loss function expresses the effect of: enhancing the effect of the better modal feature (strong modal) of the prediction result, and suppressing the effect of the worse modal feature (weak modal) of the prediction result.

[0088] As an implementable manner, the class loss function corresponding to each modal is shown in formula (10):

[0089]

[0090] wherein, is the prediction result of the y-th class according to the m-th modal feature; M is the total number of types of modal features; r t m is the correlation between the m-th modal feature and the t-th modal feature; c is a selection variable used by the correlation; β is a hyperparameter for controlling the strength of downward weighting, and is selected by cross-validation.

[0091] In order to better explain the effect of the influence factor, the class loss function is further divided into the product of two terms:

[0092]

[0093] In formula (11), Q m is the information amount of the prediction result m .

[0094]

[0095]

[0096] In formula (11), Q m is the information amount of the prediction result m . m is the influence factor carrying the enhancement or suppression information of other modal; in formula (13), C t m indicates that the correlation between modal features selects the influence on the class loss function through the variable c, as follows:

[0097]

[0098] The specific influence mechanism explanation of the influence factor T m is given below. The loss function of each modal classifier considers two multiplication terms, i.e. Q m and T m . Q m adopts an entropy loss form, which describes the prediction confidence of the sample belonging to a specific modal. Minimizing Q m ​The classifier H m can be encouraged to reach its true prediction. m is the geometric mean of the target class prediction scores output by the other classifiers and the correlation C t . m m measures the effectiveness of the other classifiers in classifying the corresponding modality, i.e. when trying to correctly classify the input sample, each classifier takes into account the performance of the other classifiers, while this effectiveness is enhanced or counterbalanced by the correlation C t m , which is the discriminative learning, m referred to as the influence factor. All classifiers have a common goal: to maximize the overall classification performance by focusing on the different modalities prediction results.

[0099] The role of the influence factor is further explained by taking the influence effect between two modalities as an example:

[0100] If the correlation between the two modalities features is high, i.e. the modalities features have more similar classification results. If at this time the classifiers H m and both have good classification results, then 1-P y t will take a smaller value to suppress H m to get a better result through backpropagation, thereby avoiding overfitting. But in the case of high correlation, it is considered normal for H m and to get similar results, so C t m takes a larger value to counterbalance the suppression effect of 1-P y t , so as to produce a more reasonable influence factor T m . If at this time the classifier H m prediction result is good, while get poor results, 1-P y t will take a larger value to encourage H m to get a better result, thereby realizing the strong modality dominant prediction. But in the case of high correlation, it is considered abnormal for H m and to get significantly different results, indicating that the modality prediction is not confident, so C t m takes a smaller value to counterbalance the encouragement effect of 1-P y t , so as to produce a more reasonable influence factor T m .

[0101] If the correlation between two modalities is low, meaning the modal features have more dissimilar classification results, then the classifier H... m and All showed good classification results, 1-P y t It will suppress H by a small value m Backpropagation yields better results, but in cases of low correlation, H m and Obtaining similar results is considered an anomaly, therefore C t m By choosing a smaller value for c, we can enhance 1-P. y t The suppression effect of this reduces the differences in predicted expressions between modalities. If the classifier H at this time... m The prediction results were good, while The result was poor, 1-P y t It will encourage H with a larger value m To achieve a better result, thus realizing strong mode-dominated prediction. However, in cases of low correlation, H m and Obtaining significantly different results is considered normal, therefore C t m By choosing a larger value for c, we can enhance 1-P. y t This encourages the prediction of strong modes, making it more reasonable.

[0102] As the examples above demonstrate, comprehensively considering the performance of all classifiers can indirectly encourage strong modes to dominate predictions by suppressing weak modes. Simultaneously, it balances the function when correlation is high and enhances it when correlation is low, making the function more complete and reliable.

[0103] S106: Obtain the objective function based on the class loss function corresponding to each modality to guide the training of the emotion recognition model;

[0104] Specifically, the objective function is shown in formula (15):

[0105]

[0106] In equation (15), λ m H represents a single-modal classifier m The weights are used to balance the effects of all classifiers in the objective function. α is the penalty factor, ||·||² denotes L2 regularization, and W represents the learnable parameters in the neural network.

[0107] In the training phase, the learnable parameters are optimized according to formula (16):

[0108]

[0109] S107: According to the multi-modal information of the to-be-recognized object, an emotion recognition result is obtained by using the trained emotion recognition model.

[0110] To be consistent with the training, in the test phase, inference is performed by solving a discrete optimization problem to determine the most likely class label, as shown in formula (17):

[0111]

[0112] Embodiment 2

[0113] Corresponding to the above method, the embodiment of the application also provides a multi-modal emotion recognition system based on discriminative learning, comprising: a collection module, a feature extraction module, a correlation calculation module, a classifier module and a training and prediction module.

[0114] The collection module is configured to collect multi-modal information of a plurality of training objects, the multi-modal information comprising electroencephalogram signals, facial signals, speech signals and text signals; the feature extraction module is configured to input the multi-modal information into respective feature extraction networks to obtain electroencephalogram features F e , facial features F v , speech features F a and text features F t ; the correlation calculation module is configured to calculate the correlation between any two modal features using a canonical correlation analysis method; the classifier module is configured to input each modal feature into a respective corresponding single-modal classifier to obtain a single-modal prediction result; the training and prediction module is configured to use the correlation between any two modal features and each single-modal prediction result to design a class loss function corresponding to each modal; a target function is obtained according to the class loss function corresponding to each modal to guide the training of the emotion recognition model; and according to the multi-modal information of the to-be-recognized object, an emotion recognition result is obtained by using the trained emotion recognition model.

[0115] It should be noted that the system provided by the embodiment of the application is to realize the above-mentioned method embodiment, and the functions thereof can be referred to the above-mentioned method embodiment, which will not be described here again.

[0116] In order to verify the effectiveness of the application, the following experimental data is provided.

[0117] The method (CCDL, Correlation Collaborative Discriminative Learning) proposed in the application is compared with existing multi-modal emotion recognition methods (TFN, LMF, LF_DNN, MULT, CDL); wherein, the CDL is a discriminative learning method without considering correlation. Experiments are performed on each method using the same multi-modal data set MOSI, while the classifiers of the method and the comparison method are controlled to be the same, and the results are shown in Table 1 and Table 2. Table 1 is that the face and voice modal use the MLP classifier, and the text modal uses the LSTM classifier. Table 2 is that each modal uses the Transformer classifier.

[0118] Table 1 CCDL using MLP+LSTM classifier

[0119]

[0120] Table 2 CCDL using Transformer classifier

[0121]

[0122] As can be seen from the above table, the discriminative learning model (CCDL) proposed in the application has good performance on different classifiers, and compared with the existing method, the CCDL has obvious improvement in emotion recognition classification accuracy. By comparing the CDL with the CCDL, it can be known that the addition of correlation can better improve the performance of discriminative learning.

[0123] In addition, the class loss function in the CCDL method contains a beta hyperparameter, which is used to control the strength of downward weighting and is selected through cross-validation. Beta is one of the components of the class loss function and plays an important role in the performance of the method. The cross-validation results of CDL and CCDL under the Transformer classifier are shown in Figure 3 and Figure 4 respectively. Among them, the CDL has the best performance when beta is 0.05, and the CCDL has the best performance when beta is 0.25.

[0124] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A multimodal emotion recognition method based on discriminative learning, characterized in that, include: Step 1: Collect multimodal information from several training subjects, including EEG signals, facial signals, speech signals, and text signals; Step 2: Input the multimodal information into their respective feature extraction networks to obtain EEG features, facial features, speech features, and text features; Step 3: Calculate the correlation between any two modal features using canonical correlation analysis; Step 4: Input each modality feature into its corresponding single-modality classifier to obtain the single-modality prediction result; Step 5: Using the correlation between any two modal features and the prediction results of each single modality, design the class loss function corresponding to each modality as shown in formula (10): in, The prediction result for the y-th category based on the m-th modal feature; M is the total number of modal feature categories; r t m Let be the correlation between the m-th modal feature and the t-th modal feature; c is the selection variable used for the correlation; β is a hyperparameter. Step 6: Obtain the objective function based on the class loss function corresponding to each modality to guide the training of the emotion recognition model; the objective function is shown in formula (15): Where, λ m denoted by weight, α is the penalty factor, ||·||2 denotes L2 regularization, and W represents the learnable parameters in the emotion recognition model; Step 7: Based on the multimodal information of the object to be identified, obtain the emotion recognition result using the trained emotion recognition model.

2. The multimodal emotion recognition method based on discriminative learning according to claim 1, characterized in that, Step 1 specifically includes: Electroencephalogram (EEG) caps and cameras are used to simultaneously collect EEG signals and videos of the training subjects; facial and speech signals of the training subjects are extracted from the videos, and then the speech signals are converted into text signals.

3. The multimodal emotion recognition method based on discriminative learning according to claim 1, characterized in that, In step 2, wavelet transform is used to extract EEG features from EEG signals, OpenFace is used to extract facial features, LibROSA is used to extract speech features, and a pre-trained BERT-base model is used to extract text features.

4. The multimodal emotion recognition method based on discriminative learning according to claim 1, characterized in that, Step 2 also includes inputting EEG features, facial features, speech features, and text features into an LSTM network to learn the temporal dependencies in each modality feature.

5. The multimodal emotion recognition method based on discriminative learning according to claim 1, characterized in that, Step 3 specifically includes: Let X and Y represent the feature vectors of any two modal features. Construct the relationship between the two feature vectors X and Y and the corresponding projection vectors u and v to be determined according to formula (3): Among them, C XY =cov(X,Y) represents the covariance of X and Y, C XX =cov(X,X) represents the autocovariance of X, C YY =cov(Y,Y) represents the autocovariance of Y; Solving formula (3) yields u and v. Using u and v, variables X and Y are linearly combined to obtain the dimension-reduced variables X' and Y'. The correlation r between any two modal features is calculated according to formula (7): The correlation value ranges from -1 to 1. Values ​​of -1 and 1 indicate a strong correlation, while a value of 0 indicates no correlation.

6. The multimodal emotion recognition method based on discriminative learning according to claim 1, characterized in that, In step 6, during the training phase, the learnable parameters are optimized according to formula (16):

7. A multimodal emotion recognition system based on discriminative learning, characterized in that, include: The acquisition module is used to acquire multimodal information of several training subjects, including electroencephalogram (EEG) signals, facial signals, speech signals, and text signals. The feature extraction module is used to input the multimodal information into their respective feature extraction networks to obtain EEG features, facial features, speech features, and text features. The correlation calculation module is used to calculate the correlation between any two modal features using canonical correlation analysis. The classifier module is used to input the features of each modality into their respective single-modality classifiers to obtain the single-modality prediction results; The training and prediction module is used to design the class loss function corresponding to each modality by utilizing the correlation between features of any two modalities and the prediction results of each single modality; to obtain the objective function based on the class loss function corresponding to each modality to guide the training of the emotion recognition model; and to obtain the emotion recognition result using the trained emotion recognition model based on the multimodal information of the object to be recognized. The class loss function corresponding to each mode is shown in Equation (10): in, The prediction result for the y-th category based on the m-th modal feature; M is the total number of modal feature categories; r t m Let be the correlation between the m-th modal feature and the t-th modal feature; c is the selection variable used for the correlation; β is a hyperparameter. The objective function is shown in formula (15): Where, λ m Let represent the weights, α be the penalty factor, ||·||2 represent L2 regularization, and W represent the learnable parameters in the emotion recognition model.

Images