A multi-modal emotion recognition method, device, electronic device and storage medium

Abstract

The present invention relates to a multimodal emotion recognition method, device, electronic device and storage medium. The multimodal emotion recognition method of the present invention comprises obtaining an EEG signal, a facial signal and a voice signal of a subject; pre-processing the EEG signal, the facial signal and the voice signal respectively, and then inputting the EEG signal, the facial signal and the voice signal into a trained EEG emotion recognition model, a facial emotion recognition model and a voice emotion recognition model respectively to obtain the probability that the EEG signal, the facial signal and the voice signal correspond to each emotion; using an optimal weight distribution algorithm, weighting and weighted fusion are performed on the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion and the probability that the voice signal corresponds to each emotion to obtain the emotion category corresponding to the subject. The multimodal emotion recognition method of the present invention effectively improves the accuracy and robustness of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of visual intelligent processing technology, and in particular to a multimodal emotion recognition method, device, electronic device and storage medium. Background Art

[0002] Emotions, as complex psychological phenomena unique to humans, play a crucial role in our daily lives. Therefore, it's essential to explore methods for identifying human emotions. Emotions can be expressed in many ways: First, they can be expressed through external features like facial expressions, voice, and body movements. While these features, as subjective expressions of emotion, are relatively simple to acquire, they can also be easily disguised. Second, emotions can be reflected through internal physiological signals, such as electroencephalogram (EEG) and electro-oculogram (EOG). These signals are less susceptible to change due to subjective will and are therefore more objective and can more realistically reflect a person's emotions.

[0003] But whether it is external signals or internal physiological signals, these signals provide judgment basis for emotion recognition in different dimensions. It is neither rigorous nor accurate enough to use only one signal to judge people's emotions.

[0004] Early research on emotion recognition often relied on a single modality, such as facial expression, voice intonation, or EEG, as these modalities could reflect human emotions and therefore had a certain degree of feasibility. Although single-modality emotion recognition methods achieved impressive accuracy, the resulting models lacked robustness and were prone to misjudgment due to subjects posing as masqueraders, making them unsuitable for practical application.

[0005] Since different people express emotions in different ways, and the same person may express emotions in more than one way, the generalization and robustness of emotion recognition methods based on a single modality are still lacking. However, using multiple modalities to form information complementarity will greatly improve the robustness of emotion recognition methods. Therefore, in recent years, some researchers have also carried out research on multimodal emotion recognition.

[0006] At present, research related to emotion recognition has the following difficulties:

[0007] The modality selection is not comprehensive enough. Although there are a few multimodal emotion recognition methods, most of them adopt a bimodal form and are not well considered. The accuracy of emotion recognition methods is not high and the robustness is poor. Summary of the Invention

[0008] Based on this, the purpose of the present invention is to provide a multimodal emotion recognition method, device, electronic device and storage medium, which effectively improve the accuracy and robustness of emotion recognition.

[0009] In a first aspect, the present invention provides a multimodal emotion recognition method, comprising the following steps:

[0010] Obtaining the subject's EEG signal, facial signal, and voice signal;

[0011] After preprocessing the EEG signal, the facial signal, and the voice signal, respectively, the EEG signal, the facial signal, and the voice signal are input into a trained EEG emotion recognition model, a trained facial emotion recognition model, and a trained voice emotion recognition model, respectively, to obtain the probability that the EEG signal, the facial signal, and the voice signal correspond to each emotion;

[0012] Using an optimal weight distribution algorithm, weights are assigned to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion;

[0013] The weight assignment obtained by the optimal weight distribution algorithm is used to perform weighted fusion on the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion to obtain the emotion category corresponding to the subject.

[0014] Furthermore, using an optimal weight distribution algorithm, weights are assigned to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion, including:

[0015] S31: Obtain the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion as the input of the optimal weight distribution algorithm, wherein n modalities correspond to n regression models, a total of T trials are used for prediction, and the predicted average arousal score of trial t in the kth model is A tk , k∈{1, 2, 3,…, n}, t∈{1, 2, 3,…, T};

[0016] S32: Initialize weight set Initialize the minimum root mean square error RS to {0.00, 0.01, 0.02, . . . , 0.98, 0.99, 1.00} min ;

[0017] S33: In The middle loop enumerates the weights of n modes, and the weight of the kth mode is ω k When the sum of all weights is 1, the following formula is used to calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution:

[0018]

[0019] in, is the predicted arousal score obtained by multimodal fusion under the current weight distribution, k∈{1, 2, 3,…, n}, n is the total number of modalities, ω k is the weight of the kth mode, A tk is the predicted mean arousal score for trial t in the kth model;

[0020] S34: Use the following formula to calculate the root mean square error RS of T trials under the current weight distribution cut :

[0021]

[0022] Among them, RS cut is the root mean square error of T trials under the current weight distribution, is the predicted arousal score obtained by multimodal fusion under the current weight distribution; t For the real arousal score;

[0023] S35: Comparison RS cut and RS min The size relationship; when RS cut min When RS min Updated to RS cut The value of RS cut ≥RS min , then keep the current RS min ;

[0024] S36: Iterate S33-S35 until the loop enumeration ends and save RS min The corresponding weight distribution is used as the weight assignment of the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

[0025] Furthermore, the EEG emotion recognition model is a tLSTM structure, including a tree part consisting of 4 layers of LSTM units, a single LSTM unit and a fully connected layer;

[0026] The number of neurons in the LSTM units on the leaf nodes of the tree portion is the same.

[0027] Furthermore, the speech emotion recognition model is a LFCNN, comprising a parallel convolution structure part, a residual structure part, a continuous convolution part and an output part connected in sequence;

[0028] ​The parallel convolution structure part includes three parallel convolution layers, and the outputs of the three parallel convolution layers will be merged together and input into the residual structure part;

[0029] The main edge of the residual structure part includes two convolutional layers;

[0030] The continuous convolution part includes four continuous convolution layers;

[0031] The output part includes multiple dense layers, or the output part includes a dense layer and a SoftMax layer.

[0032] Furthermore, the EEG signal is preprocessed, including:

[0033] Independent component analysis is used to remove artifact signals from the original signal;

[0034] For the EEG signal obtained after artifact removal, wavelet transform is used to extract the power spectral density of the EEG signal, and power spectral density features of 5 frequency bands are extracted;

[0035] The time window for EEG segmentation was 10 seconds, and the data were expanded with an overlap rate of 50%.

[0036] Furthermore, the facial signal is preprocessed, including:

[0037] Perform face detection on the input image, that is, remove the parts irrelevant to the face and only retain the face image;

[0038] The detected face image is subjected to face alignment, geometric normalization and grayscale normalization to obtain a processed face signal.

[0039] Furthermore, the speech signal is preprocessed, including:

[0040] Use the following formula to convert the original frequency f of the speech signal into Mel frequency F mel :

[0041]

[0042] The Mel frequency spectrum feature corresponding to the speech signal is obtained according to the Mel frequency.

[0043] In a second aspect, the present invention further provides a multimodal emotion recognition device, comprising:

[0044] A signal acquisition module is used to acquire the subject's EEG signal, facial signal and voice signal;

[0045] an emotion probability calculation module, configured to pre-process the EEG signal, the facial signal, and the voice signal, respectively, and then input the EEG signal, the facial signal, and the voice signal into a trained EEG emotion recognition model, a trained facial emotion recognition model, and a trained voice emotion recognition model, respectively, to obtain the probability that the EEG signal, the facial signal, and the voice signal correspond to each emotion;

[0046] A weight assignment module, configured to use an optimal weight distribution algorithm to assign weights to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion;

[0047] The emotion category output module is used to use the weight assignment obtained by the optimal weight distribution algorithm to weightedly fuse the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion to obtain the emotion category corresponding to the subject.

[0048] In a third aspect, the present invention further provides an electronic device, comprising:

[0049] at least one memory and at least one processor;

[0050] The memory is used to store one or more programs;

[0051] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a multimodal emotion recognition method as described in any one of the first aspects of the present invention.

[0052] In a fourth aspect, the present invention further provides a computer-readable storage medium,

[0053] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal emotion recognition method as described in any one of the first aspects of the present invention are implemented.

[0054] The present invention provides a multimodal emotion recognition method, device, electronic device and storage medium, which have been fully verified in experiments on the CK+ dataset (including facial data), the EMO-DB dataset (including voice data) and the MAHNOB-HCI dataset (including EEG, voice and facial data). The results show that the method proposed in this patent effectively improves the accuracy and robustness of emotion recognition.

[0055] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1A schematic diagram of the overall workflow of a multimodal emotion recognition method provided by the present invention;

[0057] Figure 2 A schematic diagram of the steps of a multimodal emotion recognition method provided by the present invention;

[0058] Figure 3 Schematic diagram of the optimal weight distribution algorithm used in one embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the tLSTM structure for EEG emotion recognition used in one embodiment of the present invention;

[0060] Figure 5 Schematic diagram showing the structure comparison of the Ghost bottleneck before and after improvement used in one embodiment of the present invention;

[0061] Figure 6 Schematic diagram of the LFCNN structure for speech emotion recognition used in one embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of a confusion matrix obtained by training on the CK+ dataset in one embodiment of the present invention;

[0063] Figure 8 This is a schematic structural diagram of a multimodal emotion recognition device provided by the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0065] It should be clear that the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0066] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0067] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0068] In addition, in this application, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0069] In response to the problems in the background technology, the present invention proposes an overall workflow for multimodal emotion recognition. Figure 1 As shown in the figure, the subject first wears a brain-computer interface device and faces a high-definition camera with voice capture. This allows us to capture EEG signals, facial signals, and voice signals during the experiment. After collecting the raw signals, we need to preprocess and extract features before feeding them into the trained model for prediction. The three modalities are fed into different emotion prediction models. Once all recognition results are obtained, they are weighted and fused using decision-level fusion to produce the final fused recognition result.

[0070] Based on the above signal acquisition device, the embodiment of the present application provides a multimodal emotion recognition method, such as Figure 2 As shown, the method includes the following steps:

[0071] S01: Obtain the subject's EEG signal, facial signal, and voice signal.

[0072] S02: After preprocessing the EEG signal, the facial signal and the voice signal respectively, the EEG signal, the facial signal and the voice signal are input into the trained EEG emotion recognition model, the facial emotion recognition model and the voice emotion recognition model respectively to obtain the probability that the EEG signal, the facial signal and the voice signal correspond to each emotion.

[0073] S03: Using an optimal weight distribution algorithm, weights are assigned to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

[0074] Preferably, Figure 3 As shown in Figure 2, the optimal weight distribution algorithm includes the following sub-steps:

[0075] S31: Obtain the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion as the input of the optimal weight distribution algorithm, wherein n modalities correspond to n regression models, a total of T trials are used for prediction, and the predicted average arousal score of trial t in the kth model is A tk , k∈{1, 2, 3,…, n}, t∈{1, 2, 3,…, T}.

[0076] S32: Initialize weight set Initialize the minimum root mean square error RS to {0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00} min .

[0077] Set the weight set is {0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00}, i.e. an array starting from 0.00 and ending at 1.00 with a step size of 0.01. The root mean square error (RMSE) is used as a metric to evaluate the performance of the current weight distribution. When each modality is in the best performance weight distribution, the RMSE should be the smallest, denoted as RS min In a preferred embodiment, initializing RS min =10.

[0078] S33: In The middle loop enumerates the weights of n modes, and the weight of the kth mode is ω k When the sum of all weights is 1, the following formula is used to calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution:

[0079]

[0080] in, is the predicted arousal score obtained by multimodal fusion under the current weight distribution, k∈{1, 2, 3,…, n}, n is the total number of modalities, ω k is the weight of the kth mode, A tk is the predicted mean arousal score for trial t in the kth model;

[0081] S34: Use the following formula to calculate the root mean square error RS of T trials under the current weight distribution cut :

[0082]

[0083] Among them, RS cut is the root mean square error of T trials under the current weight distribution, is the predicted arousal score obtained by multimodal fusion under the current weight distribution; t For the real arousal score;

[0084] S35: Comparison RS cut and RS min The size relationship; when RS cut min When RS min Updated to RS cut The value of RS cut ≥RS min , then keep the current RS min .

[0085] By comparing RS cut and RS min The relationship between the size of the current weight distribution is used to determine whether it has better performance. cut min When , it is believed that the current weight distribution has better performance, so RS min Updated to RS cut , save the current weight distribution. cut ≥RS min When the current weight distribution does not show better performance, there is no need to adjust RS min Regardless of the size relationship, step 1 must be executed again to enumerate the next set of weight distributions.

[0086] S36: Iterate S33-S35 until the loop enumeration ends and save RS min The corresponding weight distribution is used as the weight assignment of the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

[0087] S04: Using the weight assignment obtained by the optimal weight distribution algorithm, the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion are weightedly fused to obtain the emotion category corresponding to the subject.

[0088] ​​The present invention provides a multimodal emotion recognition method, device, electronic device, and storage medium, which effectively improve the accuracy and robustness of emotion recognition.

[0089] The optimal weight distribution algorithm used in the present invention can assign higher weights to patterns with high reliability and lower weights to patterns with low reliability, thereby making the fusion result more accurate and robust.

[0090] In a preferred embodiment, Figure 4 As shown, it includes a tree part consisting of 4 layers of LSTM units, a single LSTM unit and a fully connected layer.

[0091] In the tree, the LSTM units at the leaf nodes all have the same number of neurons to ensure consistent output shapes. The tree consists of four layers, each representing the features of a stage. The leaf nodes are located in different layers, allowing for better integration of features from each stage. The output of the LSTM units used in the tree is the output of the entire sequence, while the output of subsequent LSTM units is the output of the last hidden layer. Finally, the network outputs arousal and valence scores, respectively, through a fully connected layer.

[0092] Based on the above-mentioned EEG emotion recognition model, the EEG signal is preprocessed, including the following steps:

[0093] S11: Use independent component analysis to remove artifact signals from the original signal.

[0094] S12: For the EEG signal obtained after artifact removal, wavelet transform is used to extract the power spectrum density of the EEG signal, and power spectrum density features of 5 frequency bands are extracted.

[0095] S13: 10 seconds was used as the time window for EEG segmentation, and a 50% overlap rate was used to expand the data.

[0096] Specifically, emotion recognition based on electroencephalogram (EEG) signals generally requires four steps: EEG signal preprocessing, feature extraction, and emotion classification. Affected by technology, the signals we collect are often not pure EEG signals, but are contaminated with some artifacts unrelated to work, such as electroencephalogram and electrocardiogram. In this paper, Independent Component Analysis (ICA) is used to remove these artifacts. The EEG signals obtained after artifact removal still only contain time-domain features, while both the time-domain and frequency-domain features of EEG signals can reflect certain emotion-related information. In our research, Power Spectral Density (PSD) is adopted. The method for extracting PSD in this paper uses wavelet transform. The specific way to obtain it can be understood as follows: Let ∈ be the expansion scale parameter and τ be the translation position parameter, then the process of one-dimensional continuous wavelet transform W f (∈, τ) can be expressed as:

[0097]

[0098] where ψ represents the one-dimensional mother wavelet function, and its calculation formula is:

[0099]

[0100] In addition, the inverse transform of the continuous wavelet is defined as:

[0101]

[0102] where, C ψ The calculation formula is expressed as:

[0103]

[0104] is the Fourier transform of ψ(t).

[0105] In this paper, PSD features of 5 frequency bands are selected: theta (4Hz < f < 8Hz), slow alpha (8Hz < f < 10Hz), alpha (8Hz < f < 12Hz), beta (12Hz < f < 30Hz), and gamma (30Hz < f < 64Hz), as well as 14 electrodes (FP1, FP2, F8, FC2, FC6, T7, CZ, C4, T8, CP1, CP2, CP6, PO4, OZ) as the research objects. In addition, three pairs of symmetric electrodes (FP1-FP2, T7-T8, CP1-CP2) are also selected to expand the number of features.

[0106] In a preferred embodiment, based on the improved GhostNet face emotion recognition model as Figure 5As shown. This patent mainly improves the Ghost bottleneck structure of GhostNet in the face expression recognition model. The original Ghost bottleneck is divided into two cases with a step size of 1 and a step size of 2. The specific structure is as follows Figure 5 (a) shows that our improved Ghost bottleneck combines the characteristics of these two operations, so it can learn more comprehensive features. The improved Ghost bottleneck structure is shown in Figure 5 (b) In addition, we added several DropOut layers to the network to prevent overfitting during training.

[0107] Based on the above-mentioned facial emotion recognition model, the facial signal is preprocessed, including the following steps:

[0108] S21: Perform face detection on the input image, that is, remove the parts irrelevant to the face and only retain the face image.

[0109] S22: performing face alignment, geometric normalization, and grayscale normalization on the detected face image to obtain a processed face signal.

[0110] Specifically, the steps required for face-based emotion recognition include face detection, face image preprocessing, feature extraction, and facial expression classification. First, the input image needs to be detected, that is, the parts irrelevant to the face are removed, and only the face image is retained. Then, the detected face image is aligned, geometrically normalized, and grayscale normalized. The network model trained with such data has stronger generalization ability. The coordinates of the left and right eye centers are (x1, y1) and (x2, y2) respectively. Assuming the angle to be rotated for face alignment is θ, the calculation formula can be expressed as:

[0111]

[0112] To verify the superiority of the improved GhostNet in facial expression recognition, we used the GhostNet network before and after improvement to verify it on the CK+ facial expression dataset.

[0113] Experimental steps:

[0114] There are only 327 valid sequences in the CK+ dataset. We extracted the last three frames of each video as the training and validation dataset. Therefore, a total of 981 valid face images were extracted from the CK+ dataset, each of which is a grayscale image of size 48×48. A training process consists of 250 epochs, with a batch size of 64 and a fixed learning rate of 10. -3The Adam optimizer was trained. In the 10-fold cross-validation, the number of correctly predicted samples of the seven emotions on the test set was recorded after each fold of training. After the 10-fold cross-validation, the confusion matrix consisting of these seven emotions was obtained.

[0115] Experimental results:

[0116] The confusion matrix of 7 emotions obtained by GhostNet and our improved GhostNet on the CK+ facial expression recognition dataset is as follows Figure 7 (a) and Figure 7 (b) Experimental results show that the average accuracy of the results obtained by training with the original GhostNet can only reach 90.21%, while the improved GhostNet achieves an average accuracy of 98.27%, which is sufficient to demonstrate the effectiveness of our improved method. It should be noted that the accuracy of fear expression is only 95%, which may be related to the small amount of fear data and the similarity between the characteristics of fear expression and contempt. Nevertheless, our proposed method has also achieved advanced results in recent research. Table 1 shows a comparison with some recent studies. It can be seen from the table that our proposed improved GhostNet outperforms other classic classification models, which fully demonstrates the superiority of our proposed method.

[0117] Table 1 Comparison with some recent studies on the CK+ dataset

[0118]

[0119]

[0120] In a preferred embodiment, Figure 6 As shown in FIG, the speech emotion recognition model is LFCNN, which includes a parallel convolution structure part, a residual structure part, a continuous convolution part and an output part connected in sequence.

[0121] The parallel convolutional structure includes three parallel convolutional layers, the outputs of which are merged and fed into the residual structure. The residual structure's main edge includes two convolutional layers. The continuous convolutional structure includes four continuous convolutional layers. The output varies depending on the dataset label. When the data label is a dimensional label, the output includes multiple dense layers. When the data label is a discrete emotion, the output includes a dense layer and a SoftMax layer.

[0122] For the final part of the model, we can flexibly design it based on the label type of the training samples. When the dataset uses a dimensional model to describe emotions, this means setting the LFCNN as a multi-task regression model, using multiple dense layers to output scores for multiple dimensions. When using a discrete model, set the LFCNN as a classification task model, with the probability of each emotion output by the SoftMax layer.

[0123] Specifically, this patent uses Mel-spectrogram features for speech emotion recognition. Mel-spectrogram is a spectrum graph that converts frequencies into Mel-scale. Based on the above speech emotion recognition model, the speech signal is preprocessed, including the following sub-steps:

[0124] S31: Use the following formula to convert the original frequency f of the speech signal into Mel frequency F mel :

[0125]

[0126] S32: Obtaining a Mel spectrum feature corresponding to the speech signal according to the Mel frequency.

[0127] Experiments on EMO-DB

[0128] Experimental procedures

[0129] The speech emotion recognition method designed in this patent is verified on the EMO-DB dataset. First, the noise in the original audio file is removed using the noisereduce library, and the dataset is expanded by speed-shifting the original file. After the above operations, the librosa library is used to extract the Mel-Spectrogram features of the speech and save the data in a numpy array for us to use for model training and verification. We have 300 epochs in a training process, the batch size is set to 64, and the initial learning rate is 10 -4 The Adam optimizer is used, and the rate of error is reduced by e every 10 epochs starting from the 150th epoch. -0.10 In order to verify that our proposed method is reliable enough, the experiment was conducted using a 10-fold cross-validation method.

[0130] Experimental results

[0131] The classification results for EMO-DB achieved an average accuracy of 94.36% and an F1 score of 94.38%. Our proposed model is only 2.28MB in size, with the number of parameters shown in Table 2. Table 3 compares our work with some recent work, comparing prediction accuracy and model size. As can be seen from the table, the proposed LFCNN has an absolute advantage in terms of lightweightness and achieves higher accuracy, demonstrating the superiority of our approach.

[0132] Table 2 LFCNN parameter distribution

[0133]

[0134] Table 3 Comparison with some recent studies on the EMO-DB dataset

[0135]

[0136] Experiments on MAHNOB-HCI

[0137] We validated our proposed decision-level fusion method on the MAHNOB-HCI dataset. The experiment used a leave-one-out cross-validation method, where each subject retains one trial data as the test set, and the data from the other trials are used as the training set.

[0138] Experimental procedures

[0139] For EEG data, we used the MNE library to extract raw EEG signal features. As described in Section 2.2.1, we extracted PSD features from five frequency bands using a 10s time window, resulting in 85 features per sample. For the tLSTM model, the number of LSTM neurons in the leaf nodes of the tree was set to 96, and the number of neurons in other nodes was set to 128. Dropout between hidden layers of all LSTM units was set to 0.5. For speech data, the processing and training settings were essentially the same as those used in the EMO-DB experiments. For facial data, we used the OpenCV library to capture images every 10 frames and convert them to grayscale. Finally, we resized the images to 48×48 and saved them in numpy arrays for model training. Furthermore, for model training for facial expression recognition, we first pre-trained the model using the FER2013 dataset and then fine-tuned the model using the MAHNOB-HCI face data.

[0140] It's worth noting that experiments on MAHNOB-HCI require changing the model's output to two scores, and the loss function for model training also needs to be modified. MAHNOB-HCI describes emotions using a two-dimensional model of arousal valence, with scores ranging from 1 to 9. We categorize the scores into high (≥5) and low (<5) to evaluate our model's performance, which is also the most widely used approach in research on arousal valence emotion models. After obtaining the predicted scores for the three modalities, we perform a weighted fusion using traditional fusion methods and our proposed optimal weight distribution algorithm. Finally, the results are classified into high or low categories based on the scores and compared with the actual results to obtain the final multimodal emotion recognition accuracy.

[0141] Experimental results

[0142] Experimental results on the MAHNOB-HCI dataset demonstrate that both the tLSTM model and our proposed optimal weight distribution algorithm achieve excellent results in EEG emotion recognition and decision-level fusion. Table 4 shows the average accuracy achieved by each method. This shows that our proposed fusion method achieves high accuracy in both the arousal and valence dimensions, and that our proposed optimal weight distribution algorithm improves classification accuracy compared to the standard fusion algorithm. It is important to note that fusion results are not necessarily more accurate than a single modality. For example, the facial expression recognition accuracy of Subjects 2 and 13 is higher than that of the fusion results. This is because the fusion results comprehensively consider the recognition results of multiple modalities and are therefore adaptable to a wider range of situations. The significance of multimodal emotion recognition methods lies not only in improving recognition accuracy but also in ensuring robustness. For example, when subjects express facial expressions that differ from their true emotions, the results of multimodal fusion will not deviate significantly from the true emotions because the subjects' EEG still represents their true emotional state. Furthermore, speech emotion recognition is a very challenging task in MAHNOB-HCI because the speech signals in the dataset include not only the subject's voice but also a large amount of stimulus material, making it difficult to purify the subject's voice. This makes it difficult to achieve high recognition rates for the speech aspects of this dataset.

[0143] Table 4. Accuracy of each method on the MAHNOB-HCI dataset (%).

[0144]

[0145] The present application also provides a multimodal emotion recognition device, such as Figure 8 As shown, the multimodal emotion recognition device 400 includes:

[0146] Signal acquisition module 401, used to acquire the subject's EEG signal, facial signal and voice signal;

[0147] an emotion probability calculation module 402 for preprocessing the EEG signal, the facial signal, and the voice signal, respectively, and then inputting the EEG signal, the facial signal, and the voice signal into a trained EEG emotion recognition model, a trained facial emotion recognition model, and a trained voice emotion recognition model, respectively, to obtain the probability that the EEG signal, the facial signal, and the voice signal correspond to each emotion;

[0148] The weight assignment module 403 is used to use the optimal weight distribution algorithm to assign weights to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

[0149] The emotion category output module 404 is used to use the weight assignment obtained by the optimal weight distribution algorithm to weightedly fuse the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion to obtain the emotion category corresponding to the subject.

[0150] Preferably, the weight assignment module includes:

[0151] The input data acquisition unit is used to obtain the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion as the input of the optimal weight distribution algorithm, wherein n modalities correspond to n regression models, a total of T trials are used for prediction, and the predicted average arousal score of trial t in the kth model is A tk , k∈{1, 2, 3,…, n}, t∈{1, 2, 3,…, T};

[0152] Data initialization unit, used to initialize the weight set Initialize the minimum root mean square error RS to {0.00, 0.01, 0.02, . . . , 0.98, 0.99, 1.00} min ;

[0153] Cycle enumeration unit, used to The middle loop enumerates the weights of n modes, and the weight of the kth mode is ω k When the sum of all weights is 1, the following formula is used to calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution:

[0154]

[0155] in, is the predicted arousal score obtained by multimodal fusion under the current weight distribution, k∈{1, 2, 3,…, n}, n is the total number of modalities, ω k is the weight of the kth mode, A tk is the predicted mean arousal score for trial t in the kth model;

[0156] The root mean square error calculation unit is used to calculate the root mean square error RS of T trials under the current weight distribution using the following formula cut :

[0157]

[0158] Among them, RS cut is the root mean square error of T trials under the current weight distribution, is the predicted arousal score obtained by multimodal fusion under the current weight distribution; t For the real arousal score;

[0159] Root mean square error update unit, used to compare RS cut and RS min The size relationship; when RS cut min When RS min Updated to RS cut The value of RS cut ≥RS min , then keep the current RS min ;

[0160] Iteration unit, used to iteratively execute loop enumeration, RMS error calculation and RMS error update until the loop enumeration ends and RS is saved min The corresponding weight distribution is used as the weight assignment of the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

[0161] Preferably, the EEG emotion recognition model is a tLSTM structure, comprising a tree-like part consisting of 4 layers of LSTM units, a single LSTM unit and a fully connected layer;

[0162] The number of neurons in the LSTM units on the leaf nodes of the tree portion is the same.

[0163] Preferably, the speech emotion recognition model is a LFCNN, comprising a parallel convolution structure part, a residual structure part, a continuous convolution part and an output part connected in sequence;

[0164] The parallel convolution structure part includes three parallel convolution layers, and the outputs of the three parallel convolution layers will be merged together and input into the residual structure part; ​

[0165] The main edge of the residual structure part includes two convolutional layers;

[0166] The continuous convolution part includes four continuous convolution layers;

[0167] The output part includes multiple dense layers, or the output part includes a dense layer and a SoftMax layer.

[0168] Preferably, preprocessing the EEG signal includes:

[0169] Independent component analysis is used to remove artifact signals from the original signal;

[0170] For the EEG signal obtained after artifact removal, wavelet transform is used to extract the power spectral density of the EEG signal, and power spectral density features of 5 frequency bands are extracted;

[0171] The time window for EEG segmentation was 10 seconds, and the data were expanded with an overlap rate of 50%.

[0172] Preferably, preprocessing the facial signal includes:

[0173] Perform face detection on the input image, that is, remove the parts irrelevant to the face and only retain the face image;

[0174] The detected face image is subjected to face alignment, geometric normalization and grayscale normalization to obtain a processed face signal.

[0175] Preferably, preprocessing the speech signal includes:

[0176] Use the following formula to convert the original frequency f of the speech signal into Mel frequency F mel :

[0177]

[0178] The Mel frequency spectrum feature corresponding to the speech signal is obtained according to the Mel frequency.

[0179] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0180] An embodiment of the present application further provides an electronic device, including:

[0181] at least one memory and at least one processor;

[0182] The memory is used to store one or more programs;

[0183] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the multimodal emotion recognition method as described above.

[0184] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0185] The present application also provides a computer-readable storage medium.

[0186] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal emotion recognition method as described above are implemented.

[0187] Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0188] The present invention provides a multimodal emotion recognition method, device, electronic device and storage medium, which have been fully verified in experiments on the CK+ dataset (including facial data), the EMO-DB dataset (including voice data) and the MAHNOB-HCI dataset (including EEG, voice and facial data). The results show that the method proposed in this patent effectively improves the accuracy and robustness of emotion recognition.

[0189] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that: The following steps are involved: Obtaining the subject's EEG signal, facial signal, and voice signal; After preprocessing the EEG signal, the facial signal, and the voice signal, respectively, the EEG signal, the facial signal, and the voice signal are input into a trained EEG emotion recognition model, a trained facial emotion recognition model, and a trained voice emotion recognition model, respectively, to obtain the probability that the EEG signal, the facial signal, and the voice signal correspond to each emotion; Using an optimal weight distribution algorithm, weights are assigned to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion; Using the weight assignment obtained by the optimal weight distribution algorithm, the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion are weightedly fused to obtain the emotion category corresponding to the subject, specifically including: S31: Obtain the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion as the input of the optimal weight distribution algorithm, wherein n modalities correspond to n regression models, a total of T trials are used for prediction, and the predicted average arousal score of trial t in the kth model is A tk , k∈{1, 2, 3,…, n}, t∈{1, 2, 3,…, T}; S32: Initialize weight set is {0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00}, and the minimum root mean square error is initialized to RS min ; S33: In The middle loop enumerates the weights of n modes, and the weight of the kth mode is ω k When the sum of all weights is 1, the following formula is used to calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution: in, is the predicted arousal score obtained by multimodal fusion under the current weight distribution, k∈{1, 2, 3,…, n}, n is the total number of modalities, ω k is the weight of the kth mode, A tk is the predicted mean arousal score for trial t in the kth model; S34: Use the following formula to calculate the root mean square error RS of T trials under the current weight distribution cut : Among them, RS cut is the root mean square error of T trials under the current weight distribution, is the predicted arousal score obtained by multimodal fusion under the current weight distribution; t For the real arousal score; S35: Comparison RS cut and RS min The size relationship; when RS cut <RS min When , it is believed that the fusion effect is better under the current weight distribution, so RS min Updated to RS cut The value of RS cut ≥RS min , then keep the current RS min ; S36: Iterate S33-S35 until the loop enumeration ends and save RS min The corresponding weight distribution is used as the weight assignment of the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

2. The multimodal emotion recognition method according to claim 1, wherein: The EEG emotion recognition model is a tLSTM structure, including a tree-like part consisting of 4 layers of LSTM units, a single LSTM unit and a fully connected layer; The number of neurons in the LSTM units on the leaf nodes of the tree portion is the same.

3. The multimodal emotion recognition method according to claim 1, wherein: The speech emotion recognition model is LFCNN, which includes a parallel convolution structure part, a residual structure part, a continuous convolution part and an output part connected in sequence; The parallel convolution structure part includes three parallel convolution layers, and the outputs of the three parallel convolution layers will be merged together and input into the residual structure part; The main edge of the residual structure part includes two convolutional layers; The continuous convolution part includes four continuous convolution layers; The output part includes multiple dense layers, or the output part includes a dense layer and a SoftMax layer.

4. A multimodal emotion recognition method according to claim 1, characterized in that: Preprocessing the EEG signal includes: Independent component analysis is used to remove artifact signals from the original signal; For the EEG signal obtained after artifact removal, wavelet transform is used to extract the power spectral density of the EEG signal, and power spectral density features of 5 frequency bands are extracted; The time window for EEG segmentation was 10 seconds, and the data were expanded with an overlap rate of 50%.

5. A multimodal emotion recognition method according to claim 4, characterized in that: Preprocessing the facial signal includes: Perform face detection on the input image, that is, remove the parts irrelevant to the face and only retain the face image; The detected face image is subjected to face alignment, geometric normalization and grayscale normalization to obtain a processed face signal.

6. A multimodal emotion recognition method according to claim 1, characterized in that: Preprocessing the speech signal includes: Use the following formula to convert the original frequency f of the speech signal into Mel frequency F mel : The Mel frequency spectrum feature corresponding to the speech signal is obtained according to the Mel frequency.

7. A multimodal emotion recognition device, characterized in that: include: A signal acquisition module is used to acquire the subject's EEG signal, facial signal and voice signal; an emotion probability calculation module, configured to pre-process the EEG signal, the facial signal, and the voice signal, respectively, and then input the EEG signal, the facial signal, and the voice signal into a trained EEG emotion recognition model, a trained facial emotion recognition model, and a trained voice emotion recognition model, respectively, to obtain the probability that the EEG signal, the facial signal, and the voice signal correspond to each emotion; A weight assignment module, configured to use an optimal weight distribution algorithm to assign weights to the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion; The emotion category output module is used to use the weight assignment obtained by the optimal weight distribution algorithm to perform weighted fusion on the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion to obtain the emotion category corresponding to the subject, specifically including: S31: obtaining the probability of the EEG signal corresponding to each emotion, the probability of the facial signal corresponding to each emotion, and the probability of the voice signal corresponding to each emotion as the input of the optimal weight distribution algorithm, wherein n modes correspond to n regression models, a total of T trials are used for prediction, and the predicted average arousal score of trial t in the kth model is A tk , k∈{1, 2, 3,…, n}, t∈{1, 2, 3,…, T}; S32: Initialize weight set is {0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00}, and the minimum root mean square error is initialized to RS min ; S33: In The middle loop enumerates the weights of n modes, and the weight of the kth mode is ω k When the sum of all weights is 1, the following formula is used to calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution: in, is the predicted arousal score obtained by multimodal fusion under the current weight distribution, k∈{1, 2, 3,…, n}, n is the total number of modalities, ω k is the weight of the kth mode, A tk is the predicted mean arousal score for trial t in the kth model; S34: Use the following formula to calculate the root mean square error RS of T trials under the current weight distribution cut : Among them, RS cut is the root mean square error of T trials under the current weight distribution, is the predicted arousal score obtained by multimodal fusion under the current weight distribution; t For the real arousal score; S35: Comparison RS cut and RS min The size relationship; when RS cut <RS min When , it is believed that the fusion effect is better under the current weight distribution, so RS min Updated to RS cut The value of RS cut ≥RS min , then keep the current RS min ; S36: Iterate S33-S35 until the loop enumeration ends and save RS min The corresponding weight distribution is used as the weight assignment of the probability that the EEG signal corresponds to each emotion, the probability that the facial signal corresponds to each emotion, and the probability that the voice signal corresponds to each emotion.

8. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a multimodal emotion recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal emotion recognition method according to any one of claims 1 to 6 are implemented.

Images