Multimodal Emotion Recognition Methods, Systems, Electronic Devices and Media for People with Hearing Impairment

Abstract

This invention discloses a multimodal emotion recognition method, system, electronic device, and medium for hearing-impaired individuals, relating to the field of emotion recognition. First, it acquires the electroencephalogram (EEG) signal and facial image of the hearing-impaired individual to be identified. The EEG signal and facial image are preprocessed separately to obtain preprocessed EEG signals and preprocessed RGB facial images. Shallow features are extracted from the preprocessed EEG signals and preprocessed RGB facial images to obtain shallow features of the EEG signal and shallow features of the face. A multimodal emotion recognition model for hearing-impaired individuals is used to extract deep features from the shallow features of the EEG signal and the shallow features of the face, and these deep features are fused. Based on the fused features, the emotion of the hearing-impaired individual to be identified is determined. This invention improves the classification effect of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, and in particular to a method, system, electronic device, and medium for emotion recognition in multimodal hearing-impaired individuals. Background Technology

[0002] Emotion recognition based on electroencephalograms (EEGs) is currently a hot research topic in the field of human-computer interaction. Most current EEG-based emotion recognition research focuses on normal individuals and patients with cognitive impairments, with limited similar studies on hearing-impaired individuals. Compared to normal individuals, hearing-impaired individuals may exhibit biases in their perception of emotions. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal emotion recognition method, system, electronic device, and medium for hearing-impaired individuals, which can identify the emotions of hearing-impaired individuals and improve the accuracy of recognition.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for emotion recognition in multimodal hearing-impaired individuals includes:

[0006] Acquire the electroencephalogram (EEG) signals and facial images of the hearing-impaired person to be identified;

[0007] The EEG signal is subjected to a first preprocessing to obtain a preprocessed EEG signal; the first preprocessing includes downsampling, bandpass filtering, concave filtering and independent component analysis.

[0008] The face image is subjected to a second preprocessing step to obtain a preprocessed RGB face image; the second preprocessing step involves face localization and cropping.

[0009] Based on the preprocessed EEG signal, differential entropy feature extraction, projection and upsampling are performed to obtain shallow features of the EEG signal.

[0010] Based on the preprocessed RGB image of the face, center cropping and normalization are performed to obtain shallow facial features.

[0011] Based on the superficial features of the EEG signals and the superficial features of the face, a multimodal emotion recognition model for hearing-impaired individuals is used to predict the emotions of the individuals to be identified, yielding prediction results. The prediction results are happiness, sadness, calmness, or fear. The multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset. The training dataset includes training EEG signals and facial images of several hearing-impaired individuals, along with corresponding emotion category labels. The multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module. Both the MBConv module and the ResNet module are connected to the Transformer module.

[0012] Optionally, the EEG signal undergoes a first preprocessing step to obtain a preprocessed EEG signal, specifically including:

[0013] The EEG signal is downsampled to obtain a downsampled EEG signal;

[0014] The downsampled EEG signal is bandpass filtered to obtain the first filtered EEG signal;

[0015] The first filtered EEG signal is subjected to concave filtering to obtain the second filtered EEG signal.

[0016] Independent component analysis was performed on the second filtered EEG signal to obtain the analyzed EEG signal.

[0017] The analyzed EEG signal is divided into multiple frequency bands to obtain preprocessed EEG signal.

[0018] Optionally, the face image undergoes a second preprocessing step to obtain a preprocessed RGB face image, specifically including:

[0019] The face image is located using a face detector from the DLIB library to obtain face location information;

[0020] The face image is cropped based on the face location information to obtain a preprocessed RGB face image.

[0021] Optionally, based on the preprocessed EEG signal, differential entropy feature extraction, projection, and upsampling are performed to obtain shallow features of the EEG signal, specifically including:

[0022] Based on the preprocessed EEG signal, differential entropy features are extracted to obtain the differential entropy features of the EEG signal.

[0023] The differential entropy features of the EEG signal are projected onto a two-dimensional space to obtain a two-dimensional feature matrix of the EEG signal.

[0024] The two-dimensional feature matrix of the EEG signal is upsampled to obtain the shallow features of the EEG signal.

[0025] Optionally, based on the preprocessed RGB image of the face, center cropping and normalization are performed to obtain shallow facial features, specifically including:

[0026] The preprocessed RGB image of the face is cropped using the center cropping method to obtain the cropped RGB image of the face;

[0027] The cropped RGB image of the face is normalized to obtain shallow facial features.

[0028] Optionally, the multivariate regression tree model can be trained using a training dataset, specifically including:

[0029] The training EEG signal is subjected to a first preprocessing to obtain the preprocessed training EEG signal.

[0030] The training face image is subjected to a second preprocessing to obtain a preprocessed training face RGB image;

[0031] Based on the preprocessed training EEG signal, differential entropy feature extraction, projection and upsampling are performed to obtain shallow features of the training EEG signal.

[0032] Based on the preprocessed RGB image of the training face, center cropping and normalization are performed to obtain shallow facial features for training.

[0033] Using the shallow features of the training EEG signal and the shallow features of the training face as inputs, and the corresponding emotion classification labels as outputs, the multivariate regression tree model is trained to obtain a multimodal emotion recognition model for people with hearing impairments.

[0034] A multimodal emotion recognition system for people with hearing impairments, comprising:

[0035] The data acquisition module is used to acquire the electroencephalogram (EEG) signals and facial images of the hearing-impaired individuals to be identified.

[0036] A first preprocessing module is used to perform a first preprocessing on the EEG signal to obtain a preprocessed EEG signal; the first preprocessing includes downsampling, bandpass filtering, concave filtering and independent component analysis.

[0037] The second preprocessing module is used to perform a second preprocessing on the face image to obtain a preprocessed RGB face image; the second preprocessing is face localization and cropping.

[0038] The first feature extraction module is used to perform differential entropy feature extraction, projection and upsampling processing on the preprocessed EEG signal to obtain shallow features of the EEG signal.

[0039] The second feature extraction module is used to perform center cropping and normalization processing on the preprocessed RGB image of the face to obtain shallow facial features.

[0040] An emotion recognition module is used to predict the emotion of the hearing-impaired person to be identified based on the superficial features of the EEG signal and the superficial features of the face, using a multimodal emotion recognition model for hearing-impaired individuals, and obtain a prediction result; the prediction result is happiness, sadness, calmness, or fear; wherein, the multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset; the training dataset includes training EEG signals and training facial images of several hearing-impaired individuals, as well as corresponding emotion category labels; the multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module; the MBConv module and the ResNet module are both connected to the Transformer module.

[0041] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described multimodal emotion recognition method for hearing-impaired individuals.

[0042] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal emotion recognition method for people with hearing impairments.

[0043] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0044] This invention discloses a multimodal emotion recognition method, system, electronic device, and medium for hearing-impaired individuals. First, it acquires the electroencephalogram (EEG) signal and facial image of the hearing-impaired individual to be identified. The EEG signal and facial image are preprocessed to obtain preprocessed EEG signals and preprocessed RGB facial images. Shallow features are extracted from the preprocessed EEG signals and preprocessed RGB facial images to obtain shallow features of the EEG signal and shallow features of the face. A multimodal emotion recognition model for hearing-impaired individuals is used to extract deep features from the shallow features of the EEG signal and the shallow features of the face, and these deep features are fused. Based on the fused features, the emotion of the hearing-impaired individual to be identified is determined. This invention uses a two-layer MBConv convolutional network to remove features from the EEG signal that contribute little to the emotion recognition classification task, and uses a ResNet network to extract key information from facial expressions, thereby improving the classification performance of emotion recognition. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the multimodal emotion recognition method for hearing-impaired individuals according to the present invention;

[0047] Figure 2 This is a schematic diagram of shallow feature extraction from electroencephalogram (EEG) signals.

[0048] Figure 3 A schematic diagram of the MBConv module;

[0049] Figure 4 This is a schematic diagram of the Bottleneck layer of the residual module;

[0050] Figure 5 A schematic diagram of the Transformer module;

[0051] Figure 6 The accuracy curves for emotion recognition using different classification methods are shown.

[0052] Figure 7 This is a general framework diagram for emotion recognition methods for multimodal hearing-impaired individuals. Detailed Implementation

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The purpose of this invention is to provide a multimodal emotion recognition method, system, electronic device, and medium for hearing-impaired individuals, which can identify the emotions of hearing-impaired individuals and improve the accuracy of recognition.

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Example 1

[0057] like Figure 1 As shown, the multimodal emotion recognition method for hearing-impaired individuals provided by this invention includes:

[0058] Step 101: Obtain the EEG signals and facial images of the hearing-impaired person to be identified.

[0059] Step 102: Perform a first preprocessing on the EEG signal to obtain a preprocessed EEG signal; the first preprocessing includes downsampling, bandpass filtering, concave filtering and independent component analysis.

[0060] As an optional implementation, step 102 specifically includes:

[0061] The EEG signal is downsampled to obtain a downsampled EEG signal.

[0062] The downsampled EEG signal is bandpass filtered to obtain the first filtered EEG signal.

[0063] The first filtered EEG signal is subjected to concave filtering to obtain the second filtered EEG signal.

[0064] Independent component analysis was performed on the second filtered EEG signal to obtain the analyzed EEG signal.

[0065] The analyzed EEG signal is divided into multiple frequency bands to obtain a preprocessed EEG signal. In practical applications, the analyzed EEG signal is divided into five frequency bands: delta band (1-3 Hz), theta band (4-7 Hz), alpha band (8-13 Hz), beta band (14-30 Hz), and gamma band (31-50 Hz).

[0066] Step 103: Perform a second preprocessing on the face image to obtain a preprocessed RGB face image; the second preprocessing is face localization and cropping.

[0067] As an optional implementation, step 103 specifically includes:

[0068] The face image is located using a face detector from the DLIB library to obtain face location information.

[0069] The face image is cropped based on the face location information to obtain a preprocessed RGB face image.

[0070] Step 104: Based on the preprocessed EEG signal, perform differential entropy feature extraction, projection, and upsampling to obtain shallow features of the EEG signal.

[0071] As an optional implementation, step 104 specifically includes:

[0072] Based on the preprocessed EEG signal, differential entropy features are extracted to obtain the differential entropy features of the EEG signal.

[0073] The differential entropy features of the EEG signal are projected onto a two-dimensional space to obtain a two-dimensional feature matrix of the EEG signal.

[0074] The two-dimensional feature matrix of the EEG signal is upsampled to obtain the shallow features of the EEG signal.

[0075] Step 105: Based on the preprocessed RGB image of the face, perform center cropping and normalization to obtain shallow facial features.

[0076] As an optional implementation, step 105 specifically includes:

[0077] The preprocessed RGB image of the face is cropped using the center cropping method to obtain the cropped RGB image of the face.

[0078] The cropped RGB image of the face is normalized to obtain shallow facial features.

[0079] Step 106: Based on the superficial features of the EEG signal and the superficial features of the face, a multimodal emotion recognition model for hearing-impaired individuals is used to predict the emotion of the hearing-impaired person to be identified, and a prediction result is obtained; the prediction result is happiness, sadness, calmness, or fear; wherein, the multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset; the training dataset includes training EEG signals and training facial images of several hearing-impaired individuals, as well as corresponding emotion category labels; the multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module; the MBConv module and the ResNet module are both connected to the Transformer module.

[0080] As an optional implementation, the multivariate regression tree model is trained using a training dataset, specifically including:

[0081] S1: Perform a first preprocessing on the training EEG signal to obtain a preprocessed training EEG signal.

[0082] In order to eliminate interference factors as much as possible during the extraction of EEG signals and facial expression features, the experimental data must be preprocessed before feature extraction.

[0083] Regarding electroencephalogram (EEG) signals, 64 channels of EEG signals were acquired from the brains of 15 hearing-impaired subjects using a 64-channel EEG acquisition system (SymAmps2, Neuroscan, Australia) with a sampling frequency of 1000Hz, conforming to the "10-20" international standard. Bilateral mastoid sensors TP9 and TP10 were used as reference electrodes, and the signals from the other 62 electrodes were rereferenced. The rereferenced EEG signals were downsampled from 1000Hz to 200Hz to retain most of the key information while reducing the data processing burden and improving the efficiency of data processing and analysis. A bandpass filter of 1-75Hz was used to obtain the dominant frequency band related to emotion in the downsampled EEG signals. A concave filter was used to remove 50Hz mains interference from the first filtered EEG signals. Independent component analysis (ICA) was applied to remove artifact noise signals such as electrooculogram (EOG), electromyogram (EMG), and electrocardiogram (ECG) signals from the second filtered EEG signals. Finally, the analyzed EEG signal was divided into five frequency bands: δ band 1-3Hz, θ band 4-7Hz, α band 8-13Hz, β band 14-30Hz, and γ band 31-50Hz.

[0084] S2: Perform a second preprocessing on the training face image to obtain a preprocessed training face RGB image.

[0085] In practical applications, the DLIB toolkit was used to process the image data (facial images) collected during the experiment regarding facial expressions. During data acquisition, video recording equipment was used to record the facial expressions of hearing-impaired subjects during the experiment. Since the facial expressions of hearing-impaired subjects changed slowly during the experiment, to avoid data redundancy, the first frame of each second of video was extracted to represent the subject's expression data within that second. Then, the face detector in the DLIB library was used to detect and locate the facial regions in the image data. Based on the detected face locations, the facial images were cropped, retaining only the facial regions, resulting in a 256×256 pixel RGB image of the face. This process removes background information irrelevant to the facial data, improving the accuracy of facial expression recognition.

[0086] S3: Based on the preprocessed training EEG signal, perform differential entropy feature extraction, projection, and upsampling to obtain shallow features of the training EEG signal.

[0087] EEG signal and facial expression feature extraction are important steps in multimodal emotion recognition. By extracting feature information related to emotion recognition, the classification task can be simplified and the classification effect can be improved.

[0088] Regarding EEG features, differential entropy (DE) features are extracted from the preprocessed EEG signal. The EEG signal is considered to conform to a Gaussian distribution N(μ,σ). 2 The time series of a sampled signal X with a fixed time length. The formula for calculating the differential entropy feature can be defined as:

[0089]

[0090] Where σ² is the variance of the sampled signal X (preprocessed EEG signal); μ is the expected value of the selected signal frequency band; and x is the true value of the EEG signal in the selected frequency band. In this embodiment, 310 DE features (5 frequency bands × 62 channels) were extracted from 5 frequency bands of 62 sampling channels for each hearing-impaired subject. According to the EEG electrode location map, the location information of the 62 channels was mapped into a 9×9 matrix, that is, the one-dimensional DE feature vector (310 features) was projected into a two-dimensional space (9×9), as shown below. Figure 2 As shown. The white areas represent unused channels and are filled with 0s. Therefore, for each EEG data sample, five two-dimensional feature matrices can be obtained, as follows: Figure 2 As shown. To obtain the feature information between each channel, the bilinear interpolation method was used to sample the 9×9 two-dimensional plane to 224×224, thereby obtaining as much feature information between EEG channels as possible. The resulting five two-dimensional planes were then stacked into a three-dimensional space (5×224×224).

[0091] S4: Based on the preprocessed RGB image of the training face, perform center cropping and normalization to obtain shallow facial features for training.

[0092] In practical applications, regarding facial expression features, the aforementioned facial expression images, after removing irrelevant backgrounds, were resized to 256×256 pixel RGB images. These images were then cropped to 224×224 pixels using a center-cropping method, achieving data augmentation without loss of feature information. To accelerate the convergence speed of image features during network model training, the facial expression features were normalized.

[0093]

[0094] Among them, X i (j,n f ) represents the nth sample from the jth sample of the i-th subject. f Features F i (j,n f The normalized value, max[F i (:,nf ] represents the nth sample among all samples of the i-th subject. f The maximum value of each feature, min[F i (:,n f ] represents the nth sample among all samples of the i-th subject. f The minimum value of each feature is then calculated. The feature values ​​are then transformed to a range between -1 and 1 to obtain the preprocessed facial expression features (shallow facial features used for training, 3×224×224).

[0095] S5: Using the shallow features of the training EEG signal and the shallow features of the training face as inputs, and the corresponding emotion classification labels as outputs, train the multivariate regression tree model to obtain a multimodal emotion recognition model for people with hearing impairments.

[0096] In practical applications, while traditional Transformer models have achieved excellent performance on large datasets, their performance on small datasets is significantly lower than that of Convolutional Neural Networks (ConvNets). This is because ordinary Transformer modules lack certain global information acquisition capabilities possessed by ConvNets. To combine the advantages of ConvNets and Transformers, the MRT classification algorithm model (Multivariate Regression Trees) was proposed. This model combines the high generalization ability and fast convergence speed of Convolutional Neural Networks with the large model capacity of Transformers.

[0097] (I) Extraction of deep features from electroencephalogram (EEG) signals.

[0098] In the process of extracting deep features from EEG signals, a convolutional module with the MBConv module as its core was designed, and its structure is as follows: Figure 3 As shown, before processing the input data (shallow features of EEG signals used for training) with MBConv, it undergoes convolution and normalization. This not only extracts local features from the input data, helping the model better understand its structure and content, but also adjusts the distribution of the input data, allowing the model to better adapt to different input data, thereby improving its robustness and generalization ability.

[0099] The DepthwiseSeparableConvolution layer, a crucial component of the MBConv module, plays a vital role in deep feature extraction from EEG signals. It primarily consists of two parts. One part is the DepthwiseConvolution layer, which distinguishes itself from traditional convolution by performing independent convolution operations on different channels of the input data. This involves applying an h... i ×wi ×d i The size of the tensor is processed into h i ×w i ×d j The number of parameters required for traditional convolution is: h i ×w i ×d j ×d i ×k×k, while the Depthwise Convolution layer only requires h i ×w i ×d i ×(k×k+d j The number of parameters is 10. This is because in traditional convolution operations, each convolution kernel operates on every input channel of the input data simultaneously, leading to a sharp increase in the number of parameters and computational cost in deep learning. However, in the Depthwise Convolution layer, each channel of the input data has a corresponding convolution kernel, which significantly reduces the number of parameters and computational cost of the model, making the model more lightweight and reducing training costs. Another part is Pointwise Convolution, which combines features from different channels, transforming information from all input channels into a new feature channel, thereby generating a higher-dimensional feature representation that includes all channels and improving the model's expressive power.

[0100] To enhance the expressive power of features, a Squeeze-and-Excitation (SE) module was incorporated into the design of the Depthwise Separable Convolution. This module adaptively adjusts the weights of different channels in the input features, allowing the deep learning network to focus more on important features.

[0101] (II) Extraction of facial expression features.

[0102] When extracting facial expression features, a Residual Network (ResNet) module is used. ResNet is based on the structure of a convolutional neural network, and its main idea is to introduce residual structures into the network, such as... Figure 4 The diagram shown illustrates a residual structure called Bottleneck. In traditional convolution, increasing the number of convolutional layers is necessary to improve model performance, but excessive convolutional layers can lead to gradient vanishing. Residual blocks effectively solve this problem.

[0103] Considering the relatively small size of the dataset used, using a ResNet model with more layers would lead to overfitting, affecting the performance of emotion classification. Therefore, to adapt the feature extraction network to the dataset size, a 12-layer residual network was chosen to extract facial expression features. The ResNet architecture consists of several residual blocks, each containing two convolutional layers with residual connections. Normalization and ReLU activation are used after each convolutional layer to further improve the network's performance. ResNet12 can effectively capture feature information related to emotion changes in facial expression images, perceiving subtle changes in the subject's facial expressions, including important expression-related regions such as the eyes, eyebrows, nose, and mouth. Compared to the original 50-layer ResNet, the 12-layer network has fewer layers, a smaller model size, and lower computational and storage costs.

[0104] (III) Multimodal feature fusion.

[0105] In multimodal feature fusion tasks, the Transformer is chosen as the fusion module. The Transformer is a neural network used for sequence modeling, initially applied to translation tasks in natural language processing. After continuous development, it is now widely used in computer vision, speech recognition, and many other fields. In multimodal feature fusion, the Transformer plays two main roles. First, it establishes the interaction relationships between different modalities by adaptively weighting and averaging the features of different modalities through an attention mechanism, fusing them into a single vector. This not only allows features from different modalities to influence each other but also avoids the lack of correlation between features, making the fused features more accurate and useful. Second, the Transformer can perform high-dimensional mapping and nonlinear transformations on the input features through multiple layers of self-attention and feed-forward networks, extracting high-order feature representations to better represent the multimodal features of the input and improve model performance. The model structure is as follows: Figure 5 As shown.

[0106] In this invention, the MRT model is applied. By effectively combining the convolutional layers of the convolutional neural network and the multi-head self-attention layer of the Transformer network, both deep feature extraction of multimodal information and multimodal feature fusion are achieved. Finally, the classification prediction module (i.e., the fully connected layer) is used to obtain the classification prediction results of four emotions (happiness, sadness, calmness and fear) of hearing-impaired people.

[0107] This invention employs a five-fold cross-validation method for subject-dependent experimental verification. To better verify the effectiveness of the MRT model used in this invention in multimodal emotion recognition tasks, various classification methods were tested, and the classification performance of single EEG signals, single facial expression features, and multimodal fusion features on machine learning and deep learning MRT models were tested respectively.

[0108] To verify the effectiveness of the proposed MBConv and ResNet in extracting deep features from EEG signals and facial expressions, a single-modal classification comparison experiment was designed. The classification performance of single EEG signal features in machine learning (SVM) and deep learning (MBConv+Transformer, ResNet+Transformer) was tested respectively. On the machine learning classifier, the single EEG signal classification accuracy was 51.22%. EEG signal features and facial expression features achieved average accuracies of 65.76% and 75.56% on the MBConv+Transformer and ResNet+Transformer deep learning classifiers, respectively. This result demonstrates the effectiveness of the proposed MBConv+Transformer and ResNet+Transformer deep feature extraction modules in feature extraction. Furthermore, after fusing facial expression features and EEG signal features through the MRT model, an average recognition accuracy of 80.36% was achieved in the four-class emotion classification dataset for hearing-impaired subjects. Compared to single-modal emotion classification, the accuracy improved by 14.6% and 4.8%, respectively, representing a significant improvement over single-modal classification accuracy. The classification accuracy of each subject is as follows: Figure 6 As shown in Table 1, the average accuracy rates are as follows.

[0109] Table 1. Average accuracy statistics for different methods

[0110] No. method Average accuracy 1 SVM 51.22％ 2 MBConv+Transformer 65.76％ 3 ResNet+Transformer 75.56％ 4 MRT 80.36％

[0111] The multimodal emotion recognition method for hearing-impaired individuals provided by this invention, such as... Figure 7 As shown, this invention mainly comprises three parts: preprocessing of EEG signals and facial expressions, extraction of EEG signal and facial expression features, and construction of the MRT algorithm model. This invention employs the MRT deep learning network to identify the emotions of hearing-impaired individuals through deep feature extraction of EEG signal features and facial expression features. For the input EEG signals and superficial facial expression features, the MBConv module is used in the classification model to extract deep features of the EEG signals, and the ResNet12 module is used to extract features from the facial expression images. A Transformer module is designed and used to fuse multimodal deep features and complete the emotion classification task.

[0112] This invention identifies four types of emotions—happiness, sadness, fear, and calmness—by analyzing EEG signals and facial expression information generated from watching emotion-evoked videos by 15 hearing-impaired subjects. Addressing the multimodal emotion recognition problem in hearing-impaired individuals, an MRT deep learning network was designed. This network effectively combines convolutional neural networks and Transformer networks, improving emotion classification performance. In the convolutional module, two layers of MBConv convolutional networks remove features from the EEG signals that contribute little to the emotion recognition classification task, while a ResNet network extracts key information from facial expressions, further enhancing classification accuracy. In the Transformer module, two layers of multi-head self-attention modules not only achieve feature interaction between different feature layers within a single modality (five frequency bands of the EEG signal and three RGB channels of facial expressions) but also facilitate information interaction between two different modalities. After training and testing on a multimodal dataset, good classification results were achieved.

[0113] Example 2

[0114] To implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a multimodal emotion recognition system for hearing-impaired individuals is provided below, comprising:

[0115] The data acquisition module is used to acquire the electroencephalogram (EEG) signals and facial images of the hearing-impaired person to be identified.

[0116] The first preprocessing module is used to perform a first preprocessing on the EEG signal to obtain a preprocessed EEG signal; the first preprocessing includes downsampling, bandpass filtering, concave filtering and independent component analysis.

[0117] The second preprocessing module is used to perform a second preprocessing on the face image to obtain a preprocessed RGB face image; the second preprocessing is face localization and cropping.

[0118] The first feature extraction module is used to perform differential entropy feature extraction, projection, and upsampling processing on the preprocessed EEG signal to obtain shallow features of the EEG signal.

[0119] The second feature extraction module is used to perform center cropping and normalization processing on the preprocessed RGB image of the face to obtain shallow facial features.

[0120] An emotion recognition module is used to predict the emotion of the hearing-impaired person to be identified based on the superficial features of the EEG signal and the superficial features of the face, using a multimodal emotion recognition model for hearing-impaired individuals, and obtain a prediction result; the prediction result is happiness, sadness, calmness, or fear; wherein, the multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset; the training dataset includes training EEG signals and training facial images of several hearing-impaired individuals, as well as corresponding emotion category labels; the multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module; the MBConv module and the ResNet module are both connected to the Transformer module.

[0121] Example 3

[0122] The present invention provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multimodal emotion recognition method for hearing-impaired people according to Embodiment 1.

[0123] Example 4

[0124] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal emotion recognition method for hearing-impaired individuals as described in Embodiment 1.

[0125] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0126] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for emotion recognition in multimodal hearing-impaired individuals, characterized in that, include: Acquire the electroencephalogram (EEG) signals and facial images of the hearing-impaired person to be identified; The EEG signal is subjected to a first preprocessing to obtain a preprocessed EEG signal; The first preprocessing includes downsampling, bandpass filtering, concave filtering, and independent component analysis; The face image is subjected to a second preprocessing to obtain a preprocessed RGB face image; The second preprocessing step involves face localization and cropping. Based on the preprocessed EEG signal, differential entropy feature extraction, projection and upsampling are performed to obtain shallow features of the EEG signal. Based on the preprocessed RGB image of the face, center cropping and normalization are performed to obtain shallow facial features. Based on the superficial features of the EEG signals and the superficial features of the face, a multimodal emotion recognition model for hearing-impaired individuals is used to predict the emotions of the individuals to be identified, yielding prediction results. The prediction results are happiness, sadness, calmness, or fear. The multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset. The training dataset includes training EEG signals and training facial images of several hearing-impaired individuals, along with corresponding emotion category labels. The multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module. Both the MBConv module and the ResNet module are connected to the Transformer module. The MBConv module is used to extract deep features from the EEG signals, and the ResNet12 module is used to extract features from facial expression images. The Transformer module is used to fuse the multimodal deep features and complete the emotion classification task.

2. The method for emotion recognition in multimodal hearing-impaired individuals according to claim 1, characterized in that, The EEG signal undergoes a first preprocessing step to obtain a preprocessed EEG signal, specifically including: The EEG signal is downsampled to obtain a downsampled EEG signal; The downsampled EEG signal is bandpass filtered to obtain the first filtered EEG signal; The first filtered EEG signal is subjected to concave filtering to obtain the second filtered EEG signal. Independent component analysis was performed on the second filtered EEG signal to obtain the analyzed EEG signal. The analyzed EEG signal is divided into multiple frequency bands to obtain preprocessed EEG signal.

3. The method for emotion recognition in multimodal hearing-impaired individuals according to claim 1, characterized in that, The face image undergoes a second preprocessing step to obtain a preprocessed RGB face image, specifically including: The face image is located using a face detector from the DLIB library to obtain face location information; The face image is cropped based on the face location information to obtain a preprocessed RGB face image.

4. The method for emotion recognition in multimodal hearing-impaired individuals according to claim 1, characterized in that, Based on the preprocessed EEG signal, differential entropy feature extraction, projection, and upsampling are performed to obtain shallow features of the EEG signal, specifically including: Based on the preprocessed EEG signal, differential entropy features are extracted to obtain the differential entropy features of the EEG signal. The differential entropy features of the EEG signal are projected onto a two-dimensional space to obtain a two-dimensional feature matrix of the EEG signal. The two-dimensional feature matrix of the EEG signal is upsampled to obtain the shallow features of the EEG signal.

5. The method for emotion recognition in multimodal hearing-impaired individuals according to claim 1, characterized in that, Based on the preprocessed RGB image of the face, center cropping and normalization are performed to obtain shallow facial features, specifically including: The preprocessed RGB image of the face is cropped using the center cropping method to obtain the cropped RGB image of the face; The cropped RGB image of the face is normalized to obtain shallow facial features.

6. The method for emotion recognition in multimodal hearing-impaired individuals according to claim 1, characterized in that, Training the multivariate regression tree model using the training dataset includes: The training EEG signal is subjected to a first preprocessing to obtain the preprocessed training EEG signal. The training face image is subjected to a second preprocessing to obtain a preprocessed training face RGB image; Based on the preprocessed training EEG signal, differential entropy feature extraction, projection and upsampling are performed to obtain shallow features of the training EEG signal. Based on the preprocessed RGB image of the training face, center cropping and normalization are performed to obtain shallow facial features for training. Using the shallow features of the training EEG signal and the shallow features of the training face as inputs, and the corresponding emotion classification labels as outputs, the multivariate regression tree model is trained to obtain a multimodal emotion recognition model for people with hearing impairments.

7. A multimodal emotion recognition system for people with hearing impairments, characterized in that, include: The data acquisition module is used to acquire the electroencephalogram (EEG) signals and facial images of the hearing-impaired individuals to be identified. The first preprocessing module is used to perform a first preprocessing on the EEG signal to obtain a preprocessed EEG signal. The first preprocessing includes downsampling, bandpass filtering, concave filtering, and independent component analysis; The second preprocessing module is used to perform a second preprocessing on the face image to obtain a preprocessed RGB face image; The second preprocessing step involves face localization and cropping. The first feature extraction module is used to perform differential entropy feature extraction, projection and upsampling processing on the preprocessed EEG signal to obtain shallow features of the EEG signal. The second feature extraction module is used to perform center cropping and normalization processing on the preprocessed RGB image of the face to obtain shallow facial features. An emotion recognition module is used to predict the emotion of the hearing-impaired person to be identified based on the superficial features of the EEG signal and the superficial features of the face, using a multimodal emotion recognition model for hearing-impaired individuals. The predicted result is happiness, sadness, calmness, or fear. The multimodal emotion recognition model for hearing-impaired individuals is determined by training a multivariate regression tree model using a training dataset. The training dataset includes EEG signals and facial images of several hearing-impaired individuals for training, along with corresponding emotion category labels. The multivariate regression tree model includes an MBConv module, a ResNet module, and a Transformer module. Both the MBConv module and the ResNet module are connected to the Transformer module. The MBConv module is used to extract deep features from the EEG signal, and the ResNet12 module is used to extract features from facial expression images. The Transformer module is used to fuse the multimodal deep features and complete the emotion classification task.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the multimodal hearing-impaired person emotion recognition method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multimodal emotion recognition method for hearing-impaired individuals as described in any one of claims 1-6.

Images