Emotion recognition method based on electroencephalogram and expression data

Abstract

The invention discloses an emotion recognition method based on electroencephalogram (EEG) and expression data, which combines fusion analysis of electroencephalogram (EEG) signals and facial expression data, and provides an innovative solution for the complexity and accuracy of emotion recognition. The method comprises the following specific implementation steps: 1, finding an optimal frame in an expression video by using multi-instance learning, extracting global visual features by adopting Swin Transform, and realizing dimension reduction and balance representation of cross-modal information through a cross attention mechanism; 2, preprocessing the electroencephalogram signal by using band-pass filtering and a fast ICA algorithm, and extracting effective features by using PSD; and 3, designing a fusion module based on Transform, combining the characteristics of the two modes, and giving full play to the complementary advantage of the modes. Experimental results show that 94.85% of classification accuracy is achieved in four types of emotion classification tasks, and compared with an existing method, the accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and pattern recognition, and relates to an emotion recognition technology of multi-modal multi-instance data, specifically an emotion recognition algorithm combining electroencephalogram signals and facial expression data. Background Art

[0002] As an important part of human daily life, emotions have a profound impact on individual behavior, decision-making processes, social interactions, and mental health. In recent years, with the rapid development of human-computer interaction technology, emotion recognition has become a research field that has attracted much attention. A large number of studies have explored methods for emotion recognition through different data modalities. Especially in the field of affective computing, it is particularly important to use multiple data modalities to identify and understand emotions. The rapid development of deep learning methods and the use of large-scale multimodal datasets (such as SEMAINE, DEAP, MAHNOB-HCI, DREAMER, and AMIGOS) have greatly promoted the progress in the field of emotion recognition. These advances have made it possible to detect and interpret human emotions with unprecedented accuracy.

[0003] Emotion recognition relies primarily on two human data modalities: visual data (e.g., images or videos) and physiological signals (e.g., electroencephalogram (EEG)). Research initially focused on vision-based approaches using facial images and videos, as reflected in previous studies. Facial expressions have been widely studied as direct and easily captured indicators of human emotions. However, common expressions in datasets used for facial emotion recognition often display exaggerated or highly recognizable features. While easily recognizable, these expressions do not represent real-life emotional expressions, which are often not expressed in exaggerated or obvious ways in real-life scenarios. In fact, people often show little or no expression in their emotional responses. This discrepancy poses a significant challenge to accurately identifying emotions in real-life scenarios and highlights a major shortcoming in the field of facial emotion recognition. In contrast, physiological signals such as electroencephalogram (EEG) provide more accurate insights into human emotions by reflecting internal responses that are instinctive and not under conscious control. Despite their potential, the application of deep learning strategies to physiological signals remains limited, mainly hindered by challenges such as hardware inconsistencies, differences in data preprocessing, and the high cost of biosensor data acquisition.

[0004] In addition to unimodal approaches, other recent inventive studies have highlighted the advantages of multimodal affective computing techniques, which integrate visual and physiological data to achieve more efficient emotion recognition. The unique advantages of each modality can complement each other, and their fusion improves the quality of extracted features, thereby providing a more comprehensive representation of human emotions. This significantly improves the accuracy of emotion recognition compared to unimodal methods. In addition, in multimodal datasets, facial expression data is usually derived from facial videos of participants watching stimulus videos, and their expressions are more natural and subtle. Compared with the exaggerated expressions commonly seen in unimodal datasets, this makes facial expressions in multimodal studies closer to real-life scenarios.

[0005] Although the value of EEG and facial expressions in multimodal emotion recognition has been proven, there are still huge research gaps in this field and a lot of research opportunities remain to be explored. Current research focuses more on independent feature extraction of each modality, ignoring the key task of modality fusion. Many methods rely on simple methods such as feature splicing or decision tree-based fusion techniques. However, EEG and facial expressions represent two essentially different modalities. This simple fusion strategy often leads to the loss of valuable information and fails to fully utilize the complementary features extracted from the two modalities. The present invention believes that these limitations are an important reason why existing models have failed to fully realize their potential, and future research should be based on more complex and comprehensive fusion methods.

[0006] It is worth noting that most previous studies have focused more on facial expression analysis due to the inherently higher complexity of visual information compared to EEG signals. However, the present invention argues that facial expressions in multimodal datasets are typically more limited and neutral, which is closer to real-life conditions, where it is particularly difficult to distinguish subtle emotions. In this context, EEG signals play an irreplaceable role as they are able to capture underlying emotional states that are difficult to reflect in facial expressions. This highlights the equal importance of both modalities and underscores the criticality of an effective multimodal fusion strategy. Unfortunately, previous studies have generally overlooked this point, using overly simplistic fusion techniques that fail to effectively combine these two essentially different types of data.

[0007] In the feature extraction process of facial expression modality, the field mainly adopts CNN-based architecture, which usually only captures local features rather than global features. In addition, research often relies on single-frame facial expression images rather than multi-frame time series images to evaluate emotions, thus failing to fully utilize the available information. These two limitations reduce the quality of visual feature representation and ultimately affect the performance of emotion classification.

[0008] Data alignment has always been a key difficulty in multimodal research. Most inventions and studies focus on using a frame of image to align a segment of EEG signal (generally 3 seconds). Only a few inventions and studies realize that using a frame of image selected within 3 seconds cannot fully represent the rich expression information in these 3 seconds. This leads to their inability to fully utilize the existing information and extract the optimal expression features, which ultimately affects the effect of emotion classification.

[0009] In summary, in the field of multimodal emotion recognition combining facial expressions and EEG signals, existing research has the following limitations:

[0010] 1. Most inventions and research focus on feature extraction of a single modality, while the exploration of multimodal fusion strategies is limited. Usually, relatively simple and rough fusion methods are adopted, which fail to give full play to the complementary advantages of different modalities;

[0011] 2. Most inventions and research focus on using one frame of an image to align a segment of an EEG signal, which loses information;

[0012] 3. The feature extraction process of facial expression modality mainly relies on the CNN-based model architecture. Such methods usually extract local features rather than global features, which reduces the quality of visual feature representation and thus affects the performance of emotion classification;

[0013] 4. Despite the huge potential, the number of inventions in this field is still insufficient, and there are few studies specifically focusing on the fusion of facial expressions and EEG signals for emotion recognition, leaving a huge space.

[0014] To address these challenges, this paper proposes a network architecture that can more effectively extract facial expression features while achieving a more balanced and synergistic learning process between facial expressions and EEG modalities. The method includes an EEG preprocessing module, a facial feature extraction and balancing module, a Transformer-based fusion module, and a classifier. The main contributions of this paper are as follows:

[0015] 1. This paper proposes a Transformer-based modality fusion method that more effectively integrates visual and physiological modalities, overcoming the limitations of previous simple fusion methods. At the same time, a cross-attention mechanism is used to balance the features of facial images and adjust the information flow from the expression modality to ensure a more comprehensive and efficient fusion of the features of the two modalities.

[0016] 2. The present invention adopts a multi-instance learning method to extract useful information from multiple time series of facial expression images, fully representing the visual modality, so that the model can learn information from multiple expression images. At the same time, it overcomes the problems of overfitting and imbalance between modalities caused by these newly added information.

[0017] 3. The present invention uses the pre-trained Swin Transformer fine-tuned on the emotion recognition dataset to extract high-quality features. The method of the present invention is demonstrated to be more advanced than CNN and Vision Transformer.

[0018] 4. The present invention finally achieved a four-category accuracy rate of 94.85% on the well-known multimodal dataset DEAP, which is better than other inventions and research in this field. Summary of the invention

[0019] As mentioned above, many existing methods ignore the importance of modal fusion, relying only on simple concatenation or decision tree-based methods, and fail to fully utilize the complementary characteristics between different modalities. Transformer is an emerging neural network architecture that was originally used for machine translation tasks and has achieved remarkable success in the field of natural language processing (NLP). However, its research and application in emotion recognition tasks involving EEG and facial modalities are still insufficient. In the present invention, a Transformer-based framework is proposed to achieve efficient cross-modal fusion. Figure 1 As shown, the architecture consists of four main modules: (1) EEG preprocessing and feature extraction module, (2) multi-instance facial feature extraction and balancing module, (3) modality fusion module, and (4) classification module.

[0020] In the EEG preprocessing and feature extraction module, the noise and artifacts in the EEG signal are filtered to ensure the high quality of the input data. In the multi-instance facial feature extraction and balancing module, the pre-trained SwinTransformer is used to extract features from facial expression images. Subsequently, the MIL (multiple instance learning) method is introduced to better represent facial expressions by selecting the most representative frames. In order to further reduce the feature dimension and highlight the key information, the cross-attention mechanism is applied to the output of SwinTransformer. In the modal fusion module, its input comes from the output of the first two modules. It embeds the EEG and facial features by position and modality, and inputs them into the Transformer to achieve cross-modal feature-level fusion. Finally, in the classification module, the fused features contain frequency, time and spatial information, and the final emotion classification is completed through a fully connected classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Overall structure diagram of the invention

[0022] Figure 2 The specific internal structure of the cross attention module of the present invention DETAILED DESCRIPTION

[0023] 1. EEG preprocessing and feature extraction module

[0024] Since most EEG signals are concentrated in the range of 1Hz to 50Hz, a bandpass filter with a range of 1Hz to 50Hz is selected. This filtering process has a dual function: on the one hand, it eliminates low-frequency baseline drift, electrocardiogram (ECG) interference and other high-frequency noise; on the other hand, it effectively removes the most significant power frequency interference (usually 50Hz in China). In addition, EEG signals overlap with electrooculogram (EOG) and electromyogram (EMG) signals in the frequency band range. Therefore, relying solely on a single bandpass filter is not enough to completely eliminate the interference of EOG and EMG. The present invention uses fast independent component analysis (fast ICA) to eliminate the interference of EOG and EMG. Its transfer function is:

[0025]

[0026] Where: H(f) is the frequency response of the filter, f is the frequency of the input signal, and f c1 is the lower cutoff frequency of the filter (1Hz), f c2 is the upper cutoff frequency of the filter (50 Hz), and r is the order of the filter, which determines the steepness of the frequency response. In the relevant research in the field of EEG, the order is generally taken as 3, which is also the case in the present invention.

[0027] EEG signals are composed of multiple time series, which correspond to different spatial positions of different electrodes on the acquisition device in the cerebral cortex. Similar to audio signals, frequency domain features are the most prominent features, so the spectrum of the signal is usually used as an analysis method. Among the frequency domain analysis methods, power spectral density (PSD) analysis is a commonly used method, and most previous inventions have also adopted this method to study epilepsy and hypnosis. The frequency features extracted by this method can effectively detect cognitive and motor tasks. In addition, the PSD method shows the highest robustness and effectiveness in accurately distinguishing unique spectral patterns extracted from motor imagery EEG signals. In the present invention, the Welch method is used to extract the power spectrum features of EEG signals. The data sequence is processed by data windowing to generate a modified periodogram. For the EEG signal x(t) of a certain channel, it is first divided into L segments, each segment contains G sampling points, then the i-th segment small sequence x i (t) is expressed as:

[0028] x i (t)=x(g+iG),0≤g <G,0≤i<L

[0029] Where iG is the starting point of the i-th sequence. Then, L data segments with a length of 2G can be obtained. The periodogram of the i-th signal for:

[0030]

[0031] Where w(t) is the window function, which is used to perform windowing on each segment of data to reduce spectrum leakage. U represents the power normalization factor, which is selected as follows:

[0032]

[0033] By averaging these modified periodograms, the Welch power spectrum can be obtained:

[0034]

[0035] EEG signals are mainly concentrated in the range of 1Hz to 50Hz and can be divided into five frequency bands. In the present invention, the above method is used to extract the PSD features of EEG signals of all channels and five frequency bands. Instead of manually selecting certain ones of the five frequency bands for PSD feature extraction, the key features are extracted through the powerful feature extraction capability of the deep learning network architecture (transformer). The present invention divides each sample into 0.25 second time segments, and calculates the sum of the values ​​of the five frequency bands in each time segment as a separate measurement for each channel. This method ensures that the extracted features cover both frequency domain and time information. For each time segment, the PSD feature is extracted separately, and the sample dimension obtained is [W, B×C], where W is the number of time segments, B is the number of frequency bands, and C is the number of channels.

[0036] The present invention takes 3 seconds as a segment of EEG signal, and the sampling frequency of the DEAP data set is 128Hz, so the length of the EEG signal is 384. In order to align with the expression picture information of length 768, the present invention uses a multi-layer perceptron (MLP), processes it through linear feature mapping and nonlinear activation function (Relu layer), and adds a Dropout regularization term. The length is increased to 4 times and then projected to 768, and the result is obtained after back propagation training. After the EEG data processing is completed, the data of this dimension will be handed over to the modal fusion module in the third step for further processing.

[0037] 2. Multi-Instance Facial Feature Extraction and Balancing Module

[0038] The present invention has significant advantages over existing inventions and studies in visual feature extraction. First, compared with the traditional CNN-based methods commonly used in the field, the present invention uses SwinTransformer fine-tuned on the emotion classification dataset as the backbone network for feature extraction. Previous studies usually align a single frame of facial image with a segment of EEG data (usually 3 seconds), assuming that a single frame of image can represent the entire video clip. However, it is difficult to determine which frame can best reflect the information in the 3-second video. To address this limitation, the present invention introduces multiple instance learning (MIL) into this field. By extracting multiple frames from the video clip and using the MIL method, the most representative Top K frames are selected. This method ensures that the representation of facial modalities is more comprehensive and detailed, and can capture richer emotional information. Finally, the present invention solves the important problem of imbalanced contributions of different modalities in the fusion stage. The way of selecting multiple frames of facial images by MIL inevitably introduces more visual feature tokens than EEG modalities, which may cause the model to focus too much on facial features and ignore EEG information during the fusion process. To alleviate this imbalance, the present invention adopts a cross-attention-based mechanism to enable the Top K frames to interact and learn from each other, and at the same time perform dimensionality reduction on them before entering the fusion module. This step ensures the balance of information between modalities.

[0039] 1.Swin feature extraction

[0040] Compared with the traditional CNN methods commonly used in this field, Swin Transformer shows stronger capabilities in visual feature extraction. By using a layered architecture and a shifted window attention mechanism, Swin Transformer is able to efficiently capture local and global dependencies. This design not only retains the fine-grained details of local features, but also integrates contextual information from larger areas, which is superior to CNN methods that rely on fixed local receptive fields and hierarchical feature aggregation. This enables Swin Transformer to extract richer and more robust features, providing a solid foundation for subsequent multimodal fusion tasks.

[0041] In this module, the present invention uses a 6-layer Swin Transformer as an image encoder, and its weights are initialized from a model pre-trained on the ImageNet data set. The pre-trained model has the ability to identify basic image elements such as contours and shapes after learning millions of images on ImageNet, and can output abstract expressions of high-level semantic features. Swin Transformer, as the backbone network for feature extraction, generates m feature tokens from the input facial image, and the dimension of each feature token is d. The present invention uses the most common image input size 224x224, and the size of a picture after Swin is 49x768, that is, d in the present invention is 768. These feature markers retain the hierarchical structure and contextual information, which is very suitable for the processing of the subsequent fusion stage.

[0042] 2. Multiple instance learning

[0043] In the classic supervised 4-class classification problem, the goal is to develop a model that, given an instance x∈R D Predict the target variable y∈0,1,2,3. However, in the context of the multiple instance learning (MIL) problem, the model receives a set of multiple instances (generally called a bag), represented by X={x1,x2,…,x q}, where each instance is independent of the others. This bag is associated with a single label Y, whose category belongs to one of the four categories, namely y∈0,1,2,3. Further assume that each instance in the bag has its own label y1,y2,…,y q , these labels reflect the overall label Y of the package to some extent, but cannot be directly equated with it.

[0044] In MIL research, the focus is usually on understanding the relationship between instances in a bag and choosing a method that can represent the bag more balanced or universally. Among them, the most common method is pooling, such as average pooling or max pooling. With the emergence of attention mechanisms, attention-based learnable pooling methods have gradually emerged, and one of the most famous methods is AMIL. AMIL proposes a simple but effective solution that greatly improves classification accuracy. It calculates the instance x1,x2,…,x through attention weights. q Assign weights a1, a2, …, a q , and then use these weights to perform weighted summation on the instances to obtain the pooling result for subsequent classification.

[0045] The present invention improves the AMIL method to make it more applicable in this field. Because AMIL directly performs weighted summation and classification after obtaining the attention weights, while the present invention's method also involves subsequent modality fusion with EEG data, it is necessary to avoid pooling all instances into one dimension at this stage. Instead, the present invention selects the instance with the highest attention weight a q The top K embeddings of form a set z to better represent the emotional state of the subject, and then this set is used for subsequent fusion. Wherein, K is a hyperparameter, indicating the number of selected images, which is 3 in the present invention (as shown in Table 1). The formula is as follows:

[0046] z={x q q∈T}

[0047] Among them, T represents the corresponding attention weight a q The index set of the top K largest values ​​is defined as follows:

[0048] T={q|a q for a1,a2,…,a q The largest K values ​​in

[0049] The qth instance x q The corresponding attention weight a a The value of is calculated by this formula:

[0050]

[0051] Where w∈R L×1 and V∈R L×M is a learnable parameter. This process enables the selected Top K images to effectively represent the information of the entire package.

[0052]

[0053]

[0054] Table 1 The impact of different multi-instance methods and the number of selected images on the classification results

[0055] 3. Cross-Attention

[0056] In multimodal fusion, the number of feature tokens generated by each modality has an important impact on the quality of the fusion process. If the number of feature tokens generated by the facial modality (denoted as m) is significantly greater than the number of feature tokens generated by the EEG modality (denoted as l), for example, a difference of more than 4 times, the Transformer used in the fusion process may focus too much on visual features and ignore the EEG signal features. Taking the DEAP dataset as an example, the 16-channel EEG will eventually occupy 16 feature tokens, that is, l is 16. The number of visual tokens is 3 7x7 feature maps output by Swin, with a total of 147 feature tokens, that is, m is 147. If ViT is used, m will be as high as (3x14x14=)588, which will lead to modality imbalance. This imbalance will hinder the overall classification performance. In order to achieve effective multimodal fusion, the difference in the number of tokens between modalities must be minimized.

[0057]

[0058] Table 2 Study on the impact of cross attention output size on classification results

[0059] In order to solve this problem, the cross-attention mechanism is used to adjust the dimension of the feature vector output by Swin Transformer. Specifically, the cross-attention mechanism compresses m visual feature markers into n learnable query tokens, where n does not exceed 3 times l and is closer to the size of l than m. As shown in Table 2, after experimental verification, the present invention obtains the best dimension m after dimensionality reduction of 64. This adjustment matches the number of visual markers with the number of markers of the EEG modality, thereby achieving a more balanced and coordinated fusion representation. In this way, it is ensured that the two modalities can be represented in a balanced manner in downstream tasks, ultimately improving the classification performance. The main steps of cross-attention are as follows: Figure 2 As shown, including:

[0060] Linear projection: The visual feature matrix output by Swin is recorded as M, which can be expressed as M∈R m×d The learnable query tag matrix is ​​denoted as N, which can be expressed as N∈R n×d The present invention adopts the usual 12 attention heads, that is, nhead = 12, then the key projection dimension Project M and N into query Q, key K and value V matrices respectively:

[0061] Q=NW Q ,K=MW K ,V=MW V

[0062] in, W V ∈Rd×d is the learnable projection matrix.

[0063] Attention calculation: Use the scaled dot product attention mechanism to calculate the similarity between the query and the key, and combine the value matrix to calculate the weighted output:

[0064]

[0065] Among them, A∈R n×d is the attention output. The output dimension is adjusted, and the initial m×d visual features are compressed to n×d. After the expression data is processed, the data of this dimension will be handed over to the modal fusion module in the third step for further processing.

[0066] 4. Ablation Experiment

[0067] The multi-instance facial feature extraction and balancing module is relatively complex, and contains three major contents proposed by the present invention: Swin of pre-trained weights, multi-instance learning, and cross-attention. Their effects on improving the accuracy should be analyzed qualitatively and quantitatively, and it is also necessary to look at the combination relationship between them. Therefore, the present invention conducted an ablation experiment to explore it, and the results are shown in Table 3. The experiment proves that the three modules of the present invention have played a very important positive role, especially the cross-attention module, which significantly improves the accuracy. At the same time, the experiment also illustrates one reason why the present invention is significantly superior to the existing inventions, that is, the addition of an excellent visual feature extraction module.

[0068] In summary, the facial feature extraction module obtains global feature representation through Swin and combines the cross-attention mechanism to achieve alignment and compression of visual markers, thereby ensuring a balanced representation between visual and EEG modalities and improving the quality of multimodal fusion and classification performance.

[0069]

[0070] Table 3 Ablation experiment explores the role of each module

[0071] 3. Fusion Module

[0072] In this framework, the modality fusion module aims to efficiently combine EEG signals and facial expression features while retaining modality-specific information. The following describes in detail the implementation methods of modality type embedding, position embedding, classification tokens, and modality fusion. For EEG signals and facial expression features, the input is defined as follows: The EEG signal obtained by the EEG preprocessing and feature extraction module is X EEG ∈R l×d The expression feature obtained by the multi-instance facial feature extraction and balancing module is X VIS ∈R n×d .

[0073] Modality type embedding: To distinguish the two modalities of EEG and facial features, this paper introduces modality type embedding. Using the traditional embedding method, some learnable vectors are attached to the respective input features. Modality type embedding provides the model with the source information of each input token, enabling Transformer to more effectively process and fuse modality-specific features. Its representation method is as follows:

[0074] X′ EEG =X EEG +E Modal,EEG ,X′ VIS =X VIS +E Modal,VIS

[0075] Among them, E Modal,EEG ∈R 1×d and E Modal,VIS ∈R 1×d is a learnable modality embedding matrix.

[0076] Position embedding: To preserve the temporal and spatial order information, position embedding is added to the token of each modality. The representation method is as follows:

[0077] X″ EEG =X′ EEG +E Pos,EEG ,X″ VIS =X′ VIS +E Pos,VIS

[0078] Among them, E Pos,EEG ∈R 1×d and E Pos,VIS ∈R 1×d is a learnable position embedding matrix.

[0079] Classification token (C): Aggregates global information across modalities and is directly connected to features for subsequent classification:

[0080] X″′ EEG =[C;X″ EEG ],X″′ VIS =[C;X″ VIS ]

[0081] Where C∈R 1×d is a learnable categorical feature vector.

[0082] Finally, the combined features of the two modalities are obtained and input into the Transformer for modal fusion, and finally the classification token is taken out as the output result of this module.

[0083] X Fusion =[X″′EEG ; X″′ VIS ]

[0084] 4. Classification Module

[0085] The ultimate task of the present invention is to classify emotions, and one way is to represent emotions with a two-dimensional model of valence-arousal (VA). The DEAP dataset labels emotions with arousal (Arousal, a) and valence (Valence, v), and its scores range from 1 to 9. Arousal reflects the degree of activation of emotions, such as from calm to excitement, while valence indicates the degree of pleasure of emotions, such as from unpleasant to pleasant. Similar to the mainstream approach in this field, the present invention uses 5 as the threshold to divide emotions into four quadrants of the valence-arousal space: high arousal and high valence (HAHV), high arousal and low valence (HALV), low arousal and high valence (LAHV), and low arousal and low valence (LALV). The classification rules are defined as follows:

[0086]

[0087] Here, a and v represent the arousal and valence scores, respectively, and y is the label of the classification task.

[0088] Finally, set the layer normalization, Dropout layer and add the fully connected layer for classification to generate the predicted label. The formula for this part is as follows:

[0089]

[0090] The fused features are passed through the fully connected layer to produce the classification result. The cross entropy loss is used as the objective function for training the label classifier, which is defined as follows:

[0091]

[0092] Where batchsize is the number of samples in a batch, b represents the number of sample categories, which is 4 in this invention, and y ji and Represents the true labels and predicted labels in a batch.

[0093] Finally, the experimental results show that the present invention achieves a classification accuracy of 94.85% in four categories of emotion classification tasks, showing a significant improvement in accuracy over existing methods.

Claims

1. An emotion recognition method based on EEG and facial expression data, characterized in that: It consists of four main modules: (1) EEG preprocessing and feature extraction module, (2) multi-instance facial feature extraction and balancing module, (3) modality fusion module, and (4) fully connected classifier; In the first module, the noise and artifacts in the EEG signal are filtered out; next, the pre-trained SwinTransformer is used to extract features from the facial expression images; then the MIL multi-instance learning method is introduced to better represent facial expressions by selecting the most representative frames; the cross-attention mechanism is applied to the output of SwinTransformer; then, the EEG and facial features are embedded and input into the Transformer to achieve cross-modal fusion; finally, the fused features contain frequency, time and space information, and the final emotion classification is completed through a fully connected classifier; The details are as follows:

1. EEG preprocessing and feature extraction module A bandpass filter with a range of 1 Hz to 50 Hz was selected; fast independent component analysis (fast ICA) was used to remove interference from EOG and EMG; its transfer function is: Where: H(f) is the frequency response of the filter, f is the frequency of the input signal, and f c1 is the lower cutoff frequency of the filter 1Hz, f c2 is the upper cutoff frequency of the filter, 50Hz, r is the order of the filter, which is 3; The EEG signal is composed of multiple time series, which correspond to different electrodes on the acquisition device located at different spatial positions in the cerebral cortex; extract the power spectrum characteristics of the EEG signal; generate a modified periodogram by windowing the data sequence; for a certain channel of EEG signal x(t), first divide it into L segments, each segment contains G sampling points, then the i-th segment of small sequence x i (t) is expressed as: x i (t)=x(g+iG),0≤g<G,0≤i<L Where iG is the starting point of the i-th sequence; then, L data segments with a length of 2G are obtained; the periodogram of the i-th signal for: Among them, w(t) is the window function, which is used to perform windowing on each segment of data to reduce spectrum leakage; U represents the normalization factor of power, which is selected as follows: By averaging these modified periodograms, the Welch power spectrum can be obtained: The EEG signal is divided into five frequency bands; each sample is divided into 0.25 second time segments, and the sum of the values ​​of the five frequency bands in each time segment is calculated as a separate measurement for each channel; the power spectral density PSD feature is extracted for each time segment separately, and the sample dimension obtained is [W, B×C], where W is the number of time segments, B is the number of frequency bands, and C is the number of channels; 3 seconds is taken as a segment of EEG signal. The sampling frequency of DEAP dataset is 128Hz, so the length of EEG signal is 384. In order to align with the expression picture information of length 768, multi-layer perceptron (MLP) is used to process it through linear feature mapping and nonlinear activation function Relu layer, and Dropout regularization term is added. Its length is increased to 4 times and then projected to 768. After back propagation training, the result is obtained. After the EEG data is processed, it is handed over to the modal fusion module in the third step for further processing.

2. Multi-Instance Facial Feature Extraction and Balancing Module 1) Swin feature extraction A 6-layer Swin Transformer is used as the image encoder, and its weights are initialized from the model pre-trained on the ImageNet dataset. Swin Transformer is used as the backbone network for feature extraction, generating m feature tokens from the input facial image, and the dimension of each feature token is d. The most common image input size is 224x224, and the size of an image after Swin is 49x768, that is, d is 768. 2) Multiple instance learning In the classic supervised 4-class classification problem, the goal is to develop a model that, given an instance x∈R D predict the target variable y∈0,1,2,3; however, in the context of the multiple instance learning (MIL) problem, the model receives a set of multiple instances called a bag, denoted by X={x1,x2,…,x q }, where each instance is independent of the others; this bag is associated with a single label Y, whose category belongs to one of the four categories, namely y∈0,1,2,3; each instance in the bag has its own label y1,y2,…,y q ; The attention-based learnable pooling method is the improved AMIL, which is as follows: the attention weights are calculated as instances x1, x2, …, x q Assign weights a1, a2, …, a q , after obtaining the attention weights, we directly perform weighted summation and classification, and select the one with the highest attention weight a q The top K embeddings form a set z, where K is a hyperparameter, which is 3; the formula is as follows: z={x q ∣q∈T} Among them, T represents the corresponding attention weight a q The index set of the top K largest values ​​is defined as follows: T={q|a q for a1,a2,…,a q The largest K values ​​in The qth instance x q The corresponding attention weight a q The value of is calculated by this formula: Where w∈R L×1 and V∈R L×M are learnable parameters; 3) Cross-Attention The cross-attention mechanism is used to adjust the dimension of the feature vector output by Swin Transformer; the cross-attention mechanism compresses m visual feature tokens into n learnable query tokens, where n does not exceed 3 times of l, and n is 64; Cross-cutting attention includes: Linear projection: The visual feature matrix output by Swin is recorded as M, expressed as M∈R m×d ; Let the learnable query tag matrix be N, which can be expressed as N∈R n×d ; Using the usual 12 attention heads, that is, nhead = 12, the key projection dimension Project M and N into query Q, key K and value V matrices respectively: Q=NW Q ,K=MW K ,V=MW V in, W V ∈R d×d is the learnable projection matrix; Attention calculation: Use the scaled dot product attention mechanism to calculate the similarity between the query and the key, and combine the value matrix to calculate the weighted output: Among them, A∈R n×d is the attention output; the output dimension is adjusted, and the initial m×d visual features are compressed to n×d; after the expression data is processed, the data of this dimension will be handed over to the modal fusion module in the third step for further processing; 3. Fusion Module For EEG signals and facial features, the input is defined as follows: The EEG signal obtained by the EEG preprocessing and feature extraction module is X EEG ∈R l×d ; The expression feature obtained by the multi-instance facial feature extraction and balancing module is X VIS ∈R n×d ; Modality type embedding: To distinguish the two modalities of EEG and facial features, modality type embedding is introduced. This is achieved by using traditional embedding methods, by attaching some learnable vectors to their respective input features. Modality type embedding provides the model with the source information of each input token, enabling Transformer to process and fuse modality-specific features more effectively. The representation method is as follows: X′ EEG =X EEG +E Modal,EEG ,X′ VIS =X VIS +E Modal,VIS Among them, E Modal,EEG ∈R 1×d and E Modal,VIS ∈R 1×d is a learnable modality embedding matrix; Position embedding: To preserve the temporal and spatial order information, position embedding is added to the token of each modality. The representation method is as follows: X″ EEG =X′ EEG +E Pos,EEG ,X″ VIS =X′ VIS +E Pos,VIS Among them, E Pos,EEG ∈R 1×d and E Pos,VIS ∈R 1×d is a learnable position embedding matrix; Classification token (C): Aggregates global information across modalities and is directly connected to features for subsequent classification: X″′ EEG =[C;X″ EEG ],X″′ VIS =[C;X″ VIS ] Where C∈R 1×d is a learnable classification feature vector; Finally, the combined features of the two modalities are obtained and input into the Transformer for modal fusion, and finally the classification token is taken out as the output result of this module; X Fusion =[X″′ EEG ;X″′ VIS ] 4. Classification Module The final task is to classify emotions, using the Valence-Arousal (VA) two-dimensional model to represent emotions. The DEAP dataset uses Arousal (a) and Valence (v) to annotate emotions, with scores ranging from 1 to 9. Arousal reflects the degree of activation of emotions, and with 5 as the threshold, emotions are divided into four quadrants in the Valence-Arousal space: high arousal high valence (HAHV), high arousal low valence (HALV), low arousal high valence (LAHV), and low arousal low valence (LALV). The classification rules are defined as follows: Here, a and v represent the arousal and valence scores, respectively, and y is the label of the classification task; Set layer normalization, Dropout layer and add fully connected layer for classification to generate prediction labels. The formula for this part is as follows: The fused features are passed through the fully connected layer to produce the classification results; the cross entropy loss is used as the objective function for training the label classifier, which is defined as follows: Where batchsize is the number of samples in a batch, b represents the number of sample categories, which is 4, and y ji and Represents the true labels and predicted labels in a batch.

Images