An Emotion Recognition Method Based on Modality Generation of EEG and Eye Movement Signals

Abstract

This invention discloses an emotion recognition method based on EEG and eye movement signals using modality generation, comprising the following steps: manual feature extraction; extraction of EEG spatial features; extraction of EEG temporal features; extraction of eye movement spatiotemporal features; feature concatenation and modality alignment; modality fusion; pre-training; and model testing. This invention introduces a modality generation pre-training task. Modality generation aims to generate a label for another modality given one modality as a condition. With the help of the pre-training mechanism, the model can learn robust and universal feature representations that adapt to various input conditions, effectively solving the accuracy decline problem caused by the missing modalities in multimodal emotion recognition, and significantly enhancing the model's adaptability to both modality-complete and modality-missing scenarios.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, and in particular to an emotion recognition method based on modality generation of electroencephalogram and eye movement signals. Background Technology

[0002] With technological advancements, the ability of computers to recognize and respond to human intentions in real time has become a key requirement for human-computer interaction (HCI). Emotion recognition (ER) refers to identifying corresponding emotional states through human behavior and physiological responses. Accurate emotion recognition plays a crucial role in research within HCI and artificial intelligence, and has broad application prospects in education, healthcare, and transportation. In healthcare, emotion recognition can monitor patients' emotional states, identify psychological risks, and support mental health interventions. In education, emotion recognition can identify students' emotional patterns, thereby providing deeper insights into student engagement, motivation, and potential learning difficulties. In transportation, emotion recognition helps analyze passenger emotions, thereby optimizing service scheduling and improving the user experience of online ride-hailing systems.

[0003] Generally, data used for emotion recognition can be divided into two main categories: First, behavioral data such as facial expressions, voice tone, and body posture. This type of data is simple to collect but easily manipulated or faked. Second, physiological signals such as electroencephalography (EEG), electrooculogram (EOG), electrocardiogram (ECG), electromyography (EMG), and galvanic skin response (GSR) directly reflect the physiological changes during emotional experiences. These signals are less susceptible to subjective interference and provide a more objective basis for assessing emotional states. Therefore, utilizing physiological signals for emotion recognition is of great significance in both research and practical applications.

[0004] However, emotion recognition based on physiological signals often faces the following challenges:

[0005] First, most existing methods rely solely on single physiological signals from electroencephalography (EEG) for emotion recognition research. However, emotion data can originate from various modalities, including speech, facial expressions, text, and physiological signals such as EEG and electrooculography (EOG). Single-modal emotion recognition can only capture some physiological or behavioral characteristics of emotions and cannot fully reflect the complexity of emotions.

[0006] Second, most existing emotion recognition models cannot simultaneously adapt to scenarios with both unimodal and multimodal inputs. Multimodal physiological signal acquisition is prone to modality loss issues. For example, during EEG and eye-tracking data acquisition, poor device contact, significant participant movement, or failure to focus on the screen can all lead to modality loss. Existing multimodal methods do not optimize for modality loss scenarios; they only achieve optimal performance when all modal data is available, failing to adapt to real-world applications with modality loss. Furthermore, traditional unimodal methods are architecturally designed only for single-modal inputs and cannot effectively handle multimodal data. Therefore, achieving optimal model performance under both modality-complete and modality-loss settings remains a key research challenge.

[0007] Third, traditional deep learning models often struggle to effectively model both the high temporal resolution of physiological signals and the spatial asymmetry associated with emotions when handling emotion recognition tasks. This is primarily because these methods typically rely on single-scale one-dimensional convolutional kernels to extract temporal features or use global spatial convolutional kernels for coarse-grained modeling of spatial information, making it difficult to simultaneously consider fine-grained temporal dynamics and spatial structural differences. Therefore, adapting both the high temporal resolution of physiological signals and the spatial asymmetry associated with emotions remains a challenge.

[0008] Fourth, most existing multimodal fusion methods focus on feature selection or weight allocation, but neglect the heterogeneity of different modal features in terms of temporal dynamics and discriminative ability. This makes it difficult for multimodal information to achieve deep complementarity and synergistic enhancement, resulting in insufficient discriminative power of the fused features. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention provides a simple algorithm and high-accuracy emotion recognition method based on modality generation of electroencephalogram and eye movement signals.

[0010] The technical solution of this invention to solve the above-mentioned technical problems is: a method for emotion recognition based on modality generation of electroencephalogram and eye movement signals, comprising the following steps:

[0011] Step 1, Manual Feature Extraction: Extract differential entropy features from the input EEG signal and preprocess the input eye movement signal;

[0012] Step 2, extracting EEG spatial features: The extracted differential entropy features are input into the asymmetric spatial layer of the emotion recognition network TSception to extract spatial features. Then, the spatial features are input into the efficient channel attention mechanism ECA to obtain EEG spatial features with channel attention attributes.

[0013] Step 3, Extracting EEG temporal features: The extracted differential entropy features are input into a bidirectional long short-term memory network integrating a self-attention mechanism. To further extract temporal dependency features, we obtain attention-enhanced temporal features; and through a fully connected layer, we integrate and transform the temporal features to obtain EEG temporal features.

[0014] Step 4, extract eye movement spatiotemporal features: extract features from the preprocessed eye movement signal to obtain discriminative eye movement spatiotemporal features;

[0015] Step 5, Feature splicing and modal alignment: The EEG spatial features and EEG temporal features are spliced ​​to obtain EEG spatiotemporal features. Modal alignment loss based on angular cosine distance is introduced as a regularization term to promote the consistency of cross-modal semantic representation of EEG signals and eye movement signals.

[0016] Step 6, Modal Fusion: The cross-modal information fusion module CMEF is used to fuse the spatiotemporal features of EEG and eye movement to obtain fused features;

[0017] Step 7, Pre-training: Perform cross-modal generative pre-training to obtain the pre-trained model;

[0018] Step 8, Model Testing: Supervised fine-tuning of the pre-trained model yields the emotion recognition model, achieving the final emotion classification.

[0019] The above-mentioned emotion recognition method based on modality generation using EEG and eye movement signals, in step 1, involves extracting differential entropy features from the EEG signals as follows:

[0020] First, a set of bandpass filters is used to decompose the EEG signal into several physiological frequency bands;

[0021] Then, the time-domain signal of each physiological frequency band is subjected to fast Fourier transform to obtain the power spectrum of the physiological frequency band.

[0022] Finally, the differential entropy characteristics are calculated from the power spectrum of each physiological frequency band. Assuming that the EEG signal follows a Gaussian distribution, the differential entropy characteristics of the EEG signal are calculated as follows:

[0023] ;

[0024] in, This indicates the input brainwave signal; This represents the average value of the signal over a time interval; Indicates standard deviation; Represents the characteristics of differential entropy; Represents pi; Represents an exponential function; Represented by constants A logarithmic function with base 0;

[0025] The process of preprocessing eye movement signals is as follows:

[0026] For eye-tracking signals, principal component analysis is used to reduce dimensionality and remove correlations between features. The process is as follows:

[0027] ;

[0028] in, The input eye movement signal; It is the square root of the inverse of the diagonal covariance matrix; The projection matrix formed by the selected principal component directions; This represents the preprocessed eye movement signal.

[0029] In the above-mentioned emotion recognition method based on modality generation using EEG and eye-tracking signals, step 2 of the emotion recognition network TSception includes an asymmetric spatial layer containing global convolutional kernels and hemispherical convolutional kernels. The global convolutional kernels extract spatial dependency features between whole-brain channels to obtain global spatial features. The hemispherical convolution kernel models the correspondence and differences between the left and right hemispheres, obtaining hemispherical spatial features. Global spatial features and hemispherical spatial characteristics Spatial features are obtained by concatenating them along the feature dimension after average pooling. The specific calculation process is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] in, Represents the global convolutional kernel weights. Represents a one-dimensional convolution kernel; For global spatial features; Indicates the weights of the hemispherical convolution kernel; It is a hemispherical space feature; For activation functions; Indicates the concatenation function; This indicates an average pooling operation; express function; Spatial features; This indicates a batch normalization operation; Represents the characteristics of differential entropy;

[0034] The obtained spatial features The input efficient channel attention mechanism, ECA, first aggregates the multi-channel feature information obtained through global average pooling, then generates channel weights through one-dimensional convolution, and finally uses... The function generates the weight ratio for each channel, and the ECA input is multiplied by the weight ratio to obtain the EEG spatial features with channel attention attributes. The specific process is as follows:

[0035] The obtained spatial features The efficient channel attention mechanism (ECA) first aggregates the multi-channel feature information obtained through global average pooling, as follows:

[0036] ;

[0037] in, , , They are respectively Total height, total width, total number of channels , , They are respectively Height index, width index, channel index; For the first The output values ​​of each channel after global average pooling;

[0038] Next, the kernel size is... The one-dimensional convolution generates channel weights as follows:

[0039] ;

[0040] in, This is the feature vector output by global average pooling. , For the first The output values ​​of each channel after global average pooling The kernel size represents the convolution kernel size. One-dimensional convolution operation, This represents the intermediate weight vector of the one-dimensional convolution output. The calculation formula is:

[0041] ;

[0042] in, Indicates the kernel size; Regarding the total number of channels The function; express The logarithm of ; Represents the offset of the linear mapping; Indicates closest Odd numbers; This is the scaling factor;

[0043] Last use The function will Mapping to the [0,1] interval, we obtain the normalized weights for each channel, and then the spatial features are... Multiplying by the channel weights yields EEG spatial features with channel attention attributes. The specific process is as follows:

[0044] ;

[0045] ;

[0046] in, for Activation function; This is the intermediate weight vector output by a one-dimensional convolution; This is the normalized channel weight vector; This represents channel-by-channel multiplication, i.e. The channel equals Channels and weights Multiply; This refers to the spatial characteristics of brain electrical activity.

[0047] In the above-mentioned emotion recognition method based on modality generation of EEG and eye movement signals, step 3 involves extracting differential entropy features. Input-integrated bidirectional long short-term memory network with self-attention mechanism The temporal features after attention enhancement were obtained. Then, a fully connected layer is used to process the temporal features. By integrating and transforming dimensions, the temporal characteristics of the EEG were obtained. The specific calculation process is as follows:

[0048] ;

[0049] ;

[0050] ;

[0051] in, It is a feature of differential entropy; This represents a bidirectional long short-term memory network; for The output sequence of hidden states; for Transpose of; Indicates temporal characteristics; To query the transformation matrix; for Transpose of; The key transformation matrix is... Value transformation matrix Transpose of; The dimension of the key vector; for Activation function; Indicates a fully connected layer; This refers to the temporal characteristics of electroencephalogram (EEG).

[0052] In the above-mentioned modality-based EEG and eye movement signal emotion recognition method, step 4 involves first designing a multi-scale one-dimensional temporal convolution kernel for eye movement signals to effectively capture the changing trend of pupil diameter, continuous fixation events, and instantaneous saccades and blinking behaviors. Subsequently, global and hemispherical convolution kernels are introduced to spatially model the binocular channels, respectively characterizing the overall correlation of eye movement signals and the spatial asymmetry between the left and right eyes. By adapting the temporal-spatial modeling method to the eye movement signals, discriminative spatiotemporal features of eye movement are finally obtained. The specific calculation process is as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] in, Indicates the first A one-dimensional temporal convolution kernel; This represents the number of one-dimensional temporal convolution kernels. This represents the preprocessed eye movement signal; For activation functions; For the first The output of a one-dimensional time convolution kernel; A temporal feature matrix is ​​formed by concatenating the outputs of one-dimensional temporal convolution kernels; For splicing operations; Indicates a fully connected layer; Represents the co-convolution kernel; This represents the differential convolution kernel.

[0057] The specific process of step 5 in the above-mentioned modality-based EEG and eye-tracking signal emotion recognition method is as follows:

[0058] First, spatial characteristics of brainwaves With EEG temporal characteristics By splicing the data, the spatiotemporal features of the brainwaves were obtained. ;

[0059] ;

[0060] in, For splicing operations;

[0061] Then, each is passed through a learnable first linear mapping layer. Second linear mapping layer Spatiotemporal characteristics of EEG Spatiotemporal characteristics of eye movements Projected onto a shared semantic space of the same dimension:

[0062] ;

[0063] in, This is the semantic vector of the EEG signal; This is the semantic vector of the eye-tracking signal; This is the first linear mapping layer; This is the second linear mapping layer;

[0064] Next, the angular cosine distance is defined as follows:

[0065] ;

[0066] in, for and Cosine similarity between them; It is the inverse cosine function; Represents pi; for and The angular cosine distance between them;

[0067] The modal alignment loss is the expected value of the angular cosine distance:

[0068] ;

[0069] in, This is the modal alignment loss; This is the expected value; Represents a pair of spatiotemporal features of brainwaves and eye movement spatiotemporal features .

[0070] The specific process of step 6 in the above-mentioned emotion recognition method based on modality generation using EEG and eye movement signals is as follows:

[0071] First, the aligned spatiotemporal features of the brainwaves With eye movement spatiotemporal features Projected onto the same dimension through independent linear layers:

[0072] ; ;

[0073] in, Linear operations on the spatiotemporal characteristics of EEG; The projected EEG characteristics; Linear operations on eye-tracking spatiotemporal features; The eye movement features after projection;

[0074] Then, and As the input to the cross-modal information fusion module CMEF, the core of CMEF is bidirectional cross-attention;

[0075] Finally, the outputs of the bidirectional cross-attention are fused, then normalized and pooled to obtain the fused features via a feedforward neural network. The process is as follows:

[0076] ;

[0077] in, Features of fusion; Indicates pooling operation; This indicates a normalization operation; It is a feedforward neural network; This refers to the brainwave's output of attention in response to eye movements; This refers to the attention output of eye movements to brainwaves.

[0078] In the above-mentioned emotion recognition method based on modality generation using EEG and eye movement signals, the specific execution process of bidirectional cross-attention in step 6 is as follows:

[0079] The projected EEG features serve as a query, while the projected eye movement features provide information:

[0080] ;

[0081] in, This refers to the brainwave's output of attention in response to eye movements; for Activation function; To query the weight matrix; This is the key weight matrix; Value weight matrix; The dimension of the key vector; This is the transpose of the matrix; The projected EEG characteristics; The eye movement features after projection;

[0082] The projected eye movement features serve as the query, while the projected EEG features provide information:

[0083] ;

[0084] in, This refers to the attention output of eye movements to brainwaves.

[0085] In the above-mentioned emotion recognition method based on EEG and eye movement signals with modality generation, step 7 involves a modality generation task designed during the pre-training phase. The modality generation task generates a labeled sequence of another modality using an autoregressive approach, with one modality as a condition.

[0086] The autoregressive approach employs an autoregressive generative framework, specifically, a fusion representation of classification labels. As an initial context representation, a lightweight head-by-head label-based EEG sequence is generated to predict EEG sequences; let the length of the EEG label sequence to be generated be... The generation process is performed recursively:

[0087] ;

[0088] ;

[0089] in, A fusion representation for classification labels; This serves as the initial context representation; For linear generation of the head; For the generated first One EEG marker; This is the hidden state from the previous step; The length of the EEG marker sequence to be generated;

[0090] The objective function of the modality generation task is the modality generation loss:

[0091] ;

[0092] in, Represents the modal generation loss; Indicates the first One sample; The number of samples; It is the first prediction based on eye movement signals. In the nth sample One EEG marker, It is the first In the nth sample One real EEG marker;

[0093] Modality alignment loss is introduced as an auxiliary supervision during the pre-training phase. With mode generation loss Together they constitute the pre-training objective:

[0094] ;

[0095] in, The overall goal of pre-training; This is the modal alignment loss; These are the weighting coefficients for the modal alignment loss.

[0096] In the above-mentioned modality-based EEG and eye-tracking signal emotion recognition method, step 8 involves supervised fine-tuning under both subject-dependent and subject-independent experimental protocols.

[0097] In the subject-dependent experiment, a five-fold cross-validation strategy was used and executed independently within each subject. Training and test data were obtained from the same subject, and the mean accuracy and standard deviation of all subjects were calculated.

[0098] In the independent experiment, leave-one-subject cross-validation was used. Each time, all data from one subject were left as the test set, and the data from all other subjects were used as the training set for fine-tuning and evaluation. After going through all subjects, the mean and standard deviation of the overall test results were used as the final performance index.

[0099] The beneficial effects of this invention are as follows:

[0100] 1. This invention models the spatial and temporal features of EEG signals in parallel, effectively extracts complementary information from eye movement signals and achieves feature fusion between the two, thereby realizing multimodal input.

[0101] 2. This invention introduces a modality generation pre-training task. Modality generation aims to generate a label for another modality given one modality as a condition. With the help of the pre-training mechanism, the model can learn robust and general feature representations that adapt to various input conditions. This effectively solves the problem of accuracy decline caused by partial modality loss in multimodal emotion recognition and significantly enhances the model's adaptability to both modality complete and modality missing scenarios.

[0102] 3. This invention draws on the multi-scale convolution concept of the EEG emotion recognition network TSception to retain its spatial asymmetry modeling ability in the EEG branch; at the same time, it applies a multi-scale one-dimensional temporal convolution architecture adapted to the characteristics of eye movement signals to eye movement feature extraction, thereby achieving joint modeling of EEG spatial asymmetry and high temporal resolution of eye movement. Attached Figure Description

[0103] Figure 1 This is the overall flowchart of the present invention.

[0104] Figure 2 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0105] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0106] like Figure 1 , Figure 2 As shown, an emotion recognition method based on modality generation using electroencephalogram (EEG) and eye movement signals includes the following steps:

[0107] Step 1, Manual Feature Extraction: Extract differential entropy features from the input EEG signal and preprocess the input eye movement signal.

[0108] In the SEED-IV dataset, EEG and eye-tracking signals were acquired simultaneously during the experiment and organized with a uniform trial order and emotion labels. Therefore, after preprocessing and feature extraction, the EEG and eye-tracking signals maintained strict alignment.

[0109] Differential entropy features are the result of frequency domain extraction and manual processing of raw EEG signals. This preprocessing method can improve model performance to some extent. This invention also preprocesses the raw eye movement signals, extracting features including pupil diameter and dispersion, blinking, fixation, and saccades. Raw eye movement signals are high-dimensional time series, containing a large amount of low-level detail but lacking direct emotion discrimination. Therefore, the preprocessed eye movement signals can more effectively characterize eye movement changes caused by emotional states.

[0110] The process of extracting differential entropy features from EEG signals is as follows:

[0111] First, a set of bandpass filters is used to decompose the EEG signal into several physiological frequency bands;

[0112] Then, the time-domain signal of each physiological frequency band is subjected to fast Fourier transform to obtain the power spectrum of the physiological frequency band.

[0113] Finally, the differential entropy characteristics are calculated from the power spectrum of each physiological frequency band. Assuming that the EEG signal follows a Gaussian distribution, the differential entropy characteristics of the EEG signal are calculated as follows:

[0114] ;

[0115] in, This indicates the input brainwave signal; This represents the average value of the signal over a time interval; Indicates standard deviation; Represents the characteristics of differential entropy; Represents pi; Represents an exponential function; Represented by constants A logarithmic function with base 0.

[0116] The process of preprocessing eye movement signals is as follows:

[0117] For eye-tracking signals, principal component analysis is used to reduce dimensionality and remove correlations between features. The process is as follows:

[0118] ;

[0119] in, The input eye movement signal; It is the square root of the inverse of the diagonal covariance matrix; The projection matrix formed by the selected principal component directions; This represents the preprocessed eye movement signal.

[0120] Step 2, Extracting EEG Spatial Features: The extracted differential entropy features are input into the asymmetric spatial layer of the emotion recognition network TSception to extract spatial features. These spatial features are then input into the efficient channel attention mechanism ECA to adaptively enhance the sensitivity of filters in the convolutional layers to emotional representations, resulting in EEG spatial features with channel attention attributes. ECA differs from traditional channel attention mechanisms; it achieves sparse connections across channels through one-dimensional convolution, maintaining low computational overhead while improving performance.

[0121] The asymmetric spatial layer of the emotion recognition network TSception contains global convolutional kernels and hemispherical convolutional kernels. The global convolutional kernels extract spatial dependency features between brain channels, thus obtaining global spatial features. The hemispherical convolution kernel models the correspondence and differences between the left and right hemispheres, obtaining hemispherical spatial features. Global spatial features and hemispherical spatial characteristics Spatial features are obtained by concatenating them along the feature dimension after average pooling. The specific calculation process is as follows:

[0122] ;

[0123] ;

[0124] ;

[0125] in, Represents the global convolutional kernel weights. Represents a one-dimensional convolution kernel; For global spatial features; Indicates the weights of the hemispherical convolution kernel; It is a hemispherical space feature; For activation functions; This represents a concatenation function, used to join multiple tensors along a specified dimension into a new tensor; This indicates an average pooling operation; express function; Spatial features; This indicates a batch normalization operation; Represents the characteristics of differential entropy;

[0126] The obtained spatial features The efficient channel attention mechanism (ECA) first aggregates the multi-channel feature information obtained through global average pooling, as follows:

[0127] ;

[0128] in, , , They are respectively Total height, total width, total number of channels , , They are respectively Height index, width index, channel index; For the first The output values ​​of each channel after global average pooling;

[0129] Next, the kernel size is... The one-dimensional convolution generates channel weights as follows:

[0130] ;

[0131] in, This is the feature vector output by global average pooling. , For the first The output values ​​of each channel after global average pooling The kernel size represents the convolution kernel size. One-dimensional convolution operation, This represents the intermediate weight vector of the one-dimensional convolution output. The size represents the coverage of local cross-channel interactions, and an adaptive method is used to set it. The value, The calculation formula is:

[0132] ;

[0133] in, Indicates the kernel size; Regarding the total number of channels The function represents It is determined by the total number of channels Calculated; express The logarithm of ; This represents the offset of the linear mapping, used for adjustment. The base value; Indicates closest For odd numbers, round the calculation result to the nearest odd number to ensure the convolution kernel size. The number of centers is always odd to ensure that the convolution operation has a symmetrical center point; This is the scaling factor, usually taken as... ,control Follow The rate of growth.

[0134] Last use The function will Mapping to the [0,1] interval, we obtain the normalized weights for each channel, and then the spatial features are... Multiplying by the channel weights yields EEG spatial features with channel attention attributes. The specific process is as follows:

[0135] ;

[0136] ;

[0137] in, for Activation function; This is the intermediate weight vector output by a one-dimensional convolution; This is the normalized channel weight vector; This represents channel-by-channel multiplication, i.e. The channel equals Channels and weights Multiply; This refers to the spatial characteristics of brain electrical activity.

[0138] Step 3, Extracting EEG temporal features: The extracted differential entropy features are input into a bidirectional long short-term memory network integrating a self-attention mechanism. To further extract temporal-dependent features, we obtain temporal features after attention enhancement; and through a fully connected layer, we integrate and transform the temporal features to obtain EEG temporal features.

[0139] Electroencephalogram (EEG) signals have the advantages of high temporal resolution and the ability to directly reflect neural activity. Their temporal features can effectively complement spatial features, thereby improving model performance. To better capture important temporal patterns in EEG signals, this invention... It integrates a self-attention mechanism to enhance the information weights of important time steps.

[0140] Extracted differential entropy features Input-integrated bidirectional long short-term memory network with self-attention mechanism The temporal features after attention enhancement were obtained. Then, a fully connected layer is used to process the temporal features. By integrating and transforming dimensions, the temporal characteristics of the EEG were obtained. The specific calculation process is as follows:

[0141] ;

[0142] ;

[0143] ;

[0144] in, It is a feature of differential entropy; This represents a bidirectional long short-term memory network. The temporal dependency is modeled from both forward and backward directions to obtain a context-aware representation of each time step; for The output sequence of hidden states; for Transpose of; Indicates temporal characteristics; To query the transformation matrix; for Transpose of; The key transformation matrix is... Value transformation matrix Transpose of; The dimension of the key vector is used to scale the dot product attention. for Activation function; Indicates a fully connected layer; This refers to the temporal characteristics of electroencephalogram (EEG).

[0145] Step 4, extract eye movement spatiotemporal features: extract features from the preprocessed eye movement signal to obtain discriminative eye movement spatiotemporal features.

[0146] First, multi-scale one-dimensional temporal convolutional kernels are designed for eye movement signals to effectively capture the changing trends of pupil diameter, continuous fixation events, and instantaneous saccades and blinking behaviors. Then, global and hemispherical convolutional kernels are introduced to spatially model the binocular channels, respectively characterizing the overall correlation of eye movement signals and the spatial asymmetry between the left and right eyes. By adapting the temporal-spatial modeling method to eye movement signals, discriminative spatiotemporal features of eye movements are finally obtained. The specific calculation process is as follows:

[0147] ;

[0148] ;

[0149] ;

[0150] in, Indicates the first A one-dimensional temporal convolution kernel; This represents the number of one-dimensional temporal convolution kernels. This represents the preprocessed eye movement signal; For activation functions; For the first The output of a one-dimensional time convolution kernel; A temporal feature matrix is ​​formed by concatenating the outputs of one-dimensional temporal convolution kernels; For splicing operations; Indicates a fully connected layer; This represents a collaborative convolution kernel, used to extract binocular synchronous change features; This represents the differential convolution kernel, which extracts binocular asymmetry features.

[0151] Step 5, Feature splicing and modality alignment: The spatial and temporal features of EEG are spliced ​​to obtain the spatiotemporal features of EEG. Although the EEG signal and eye movement signal are synchronized in time during acquisition, their features still have distribution differences in the semantic space. Therefore, modality alignment loss based on angular cosine distance is introduced as a regularization term to promote the consistency of cross-modal semantic representation of EEG signal and eye movement signal.

[0152] The specific process of step 5 is as follows:

[0153] First, spatial characteristics of brainwaves With EEG temporal characteristics By splicing the data, the spatiotemporal features of the brainwaves were obtained. ;

[0154] ;

[0155] in, For splicing operations;

[0156] Then, each is passed through a learnable first linear mapping layer. Second linear mapping layer Spatiotemporal characteristics of EEG Spatiotemporal characteristics of eye movements Projected onto a shared semantic space of the same dimension:

[0157] ;

[0158] in, This is the semantic vector of the EEG signal; This is the semantic vector of the eye-tracking signal; This is the first linear mapping layer; This is the second linear mapping layer;

[0159] Next, the angular cosine distance is defined as follows:

[0160] ;

[0161] in, for and The goal is to minimize the cosine similarity between the two pairs of samples during pre-training. It is the inverse cosine function; Represents pi; for and The angular cosine distance between them;

[0162] The modal alignment loss is the expected value of the angular cosine distance:

[0163] ;

[0164] in, This is the modal alignment loss; This is the expected value; Represents a pair of spatiotemporal features of brainwaves and eye movement spatiotemporal features .

[0165] It should be noted that the modality alignment loss serves only as an auxiliary supervision signal during the pre-training phase and does not alter the feature concatenation structure during the model's forward propagation. After pre-training, the fine-tuning phase uses only the cross-entropy loss from sentiment classification for optimization, and alignment constraints are no longer included.

[0166] Step 6, Modal Fusion: The cross-modal information fusion module CMEF is used to fuse the spatiotemporal features of EEG and eye movement to obtain fused features.

[0167] The specific process of step 6 is as follows:

[0168] First, the aligned spatiotemporal features of the brainwaves With eye movement spatiotemporal features Projected onto the same dimension through independent linear layers:

[0169] ; ;

[0170] in, Linear operations on the spatiotemporal characteristics of EEG; The projected EEG characteristics; Linear operations on eye-tracking spatiotemporal features; The eye movement features after projection;

[0171] Then, and As the input to the cross-modal information fusion module CMEF, the core of CMEF is bidirectional cross-attention;

[0172] Finally, the outputs of the bidirectional cross-attention are fused, then normalized and pooled to obtain the fused features via a feedforward neural network. The process is as follows:

[0173] ;

[0174] in, Features of fusion; Indicates pooling operation; This indicates a normalization operation; It is a feedforward neural network; This refers to the brainwave's output of attention in response to eye movements; This refers to the attention output of eye movements to brainwaves.

[0175] The specific execution process of bidirectional cross-attention is as follows:

[0176] The projected EEG features serve as a query, while the projected eye movement features provide information:

[0177] ;

[0178] in, This refers to the brainwave's output of attention in response to eye movements; for Activation function; To query the weight matrix; This is the key weight matrix; Value weight matrix; The dimension of the key vector; This is the transpose of the matrix; The projected EEG characteristics; The eye movement features after projection;

[0179] The projected eye movement features serve as the query, while the projected EEG features provide information:

[0180] ;

[0181] in, This refers to the attention output of eye movements to brainwaves.

[0182] Step 7, Pre-training: Perform cross-modal generative pre-training to obtain the pre-trained model.

[0183] Pre-training refers to the initial training of a model on data, enabling it to learn some universally applicable knowledge. In this invention, pre-training was performed on eight NVIDIA A800 GPUs for 200 training epochs. During the pre-training phase, the model was self-supervised using multimodal physiological signal data from all subjects to learn universal feature representations across modalities. Although the dataset provided emotion labels, pre-training did not introduce label supervision; its training objective was used for model parameter initialization. After pre-training, the model was fine-tuned and tested under subject-dependent and subject-independent experimental protocols, and the relevant results were obtained based on the fine-tuning phase.

[0184] A modality generation task was designed during the pre-training phase. This task, conditioned on a particular modality, generates a labeled sequence for another modality via an autoregressive approach. This task not only promotes cross-modal information interaction (e.g., generating EEG labels based on eye-tracking signals) but also strengthens the temporal dependencies within each modality through causal masking. With generative pre-training, the model can learn generalizable cross-modal representations, thus mitigating the few-shot problem.

[0185] The autoregressive approach employs an autoregressive generative framework, specifically, a fusion representation of classification labels. As an initial context representation, a lightweight head-by-head label-based EEG sequence is generated to predict EEG sequences; let the length of the EEG label sequence to be generated be... The generation process is performed recursively:

[0186] ;

[0187] ;

[0188] in, A fusion representation for classification labels; This serves as the initial context representation; For linear generation of the head; For the generated first One EEG marker; This is the hidden state from the previous step; The length of the EEG marker sequence to be generated;

[0189] The objective function of the modality generation task is the modality generation loss:

[0190] ;

[0191] in, Represents the modal generation loss; Indicates the first One sample; The number of samples; It is the first prediction based on eye movement signals. In the nth sample One EEG marker, It is the first In the nth sample One real EEG marker;

[0192] Modality alignment loss is introduced as an auxiliary supervision during the pre-training phase. With mode generation loss Together they constitute the pre-training objective:

[0193] ;

[0194] in, The overall goal of pre-training; This is the modal alignment loss; These are the weighting coefficients for the modal alignment loss.

[0195] Step 8, Model Testing: Supervised fine-tuning of the pre-trained model yields the emotion recognition model, achieving the final emotion classification.

[0196] In step 8, supervised fine-tuning is performed under both subject-dependent and subject-independent experimental protocols.

[0197] In the subject-dependent experiment, a five-fold cross-validation strategy was employed and executed independently within each subject. Specifically, training and testing data were drawn from the same subject, and the mean accuracy and standard deviation were calculated for all subjects.

[0198] In the independent experiment, leave-one-subject cross-validation was used. Each time, all data from one subject were left as the test set, and the data from all other subjects were used as the training set for fine-tuning and evaluation. After going through all subjects, the mean and standard deviation of the overall test results were used as the final performance index.

[0199] In the independent experiments with the subjects, several representative advanced methods were selected as benchmarks for comparison, including: MIND-EEG, BGAN, DGCCA-AM, JDA-RSDB, MHESA, BERN, and MACDB.

[0200] The above models are described below:

[0201] MIND-EEG (Discrete Codebook Multigranularity Integrated Network for EEG Emotion Recognition): It integrates EEG spatial information using multigranular encoders (global, intra-regional, and inter-regional) and combines them with a discrete codebook mechanism to generate a compact and generalized brain network representation for emotion recognition.

[0202] BGAN (Bipartite Graph Adversarial Network): Combining Transformer spatiotemporal feature extraction with bipartite graph adversarial alignment, it maps source and target domain samples to a shared bipartite graph structure. Through graph adversarial training, it learns domain-invariant and class-discriminative EEG representations, effectively solving the domain transfer problem in cross-subject emotion recognition.

[0203] DGCCA-AM (Deep Generalized Canonical Relevance Analysis Framework with Attention Mechanism): It integrates multiple eye-tracking features such as pupil size, event statistics, and fixation / salivation, and learns shared representations through weighted fusion using attention mechanism to achieve high-precision emotion classification.

[0204] JDA-RSDB (Multimodal Domain Adaptation Method): Jointly aligns edges and conditional distributions, refines decision boundaries through consistency between master and slave classifiers, places boundaries in low-density regions, and improves cross-conversation emotion recognition performance.

[0205] MHESA (Multidimensional Homogeneous Coding Space Alignment Method): Includes multimodal joint learning and modality knowledge transfer modules. By constructing an EEG-EM common projection space and preserving the original distribution, combined with multi-task joint optimization, it achieves stable cross-modal sentiment recognition.

[0206] BERN (Bimodal Emotion Recognition Network): It uses 3D convolution to extract EEG features, combines residual connections to extract eye-tracking features, and fuses the two through cross-modal attention to improve the accuracy of emotion recognition.

[0207] MACDB (Multi-level Alignment and Consistent Decision Boundary Multimodal Domain Adaptation Method): A three-level alignment strategy (intramodal, intermodal, and cross-domain) combined with prediction consistency constraints under adversarial perturbation forms a robust decision boundary in the EEG feature space, improving cross-subject recognition performance.

[0208] Table 1. Average accuracy of each method for emotion recognition task on the SEED-IV dataset.

[0209]

[0210] Table 1 shows the comparison results of our proposed method with other methods in independent experiments on the SEED-IV dataset. Based on the analysis in Table 1, the average accuracy of the proposed method reached 87.45%, surpassing all the comparison methods and verifying its effectiveness. Among all multimodal comparison methods, our proposed method achieved the highest accuracy, and the difference with the suboptimal method was significant, indicating that our proposed method has stronger robustness in cross-modal feature alignment and fusion.

[0211] Eye movement signals alone achieved an accuracy rate of only 63.14%, lower than the 75%–77% of EEG. When combined with EEG, the multimodal average accuracy reached 83.99%, an improvement of 10%–20% compared to single-modal recognition. This indicates that although eye movement signals are difficult to support high-precision emotion recognition independently, as a supplementary modality to EEG, they can provide rich complementary information, effectively enhance cross-modal feature expression capabilities, and significantly improve overall recognition performance.

Claims

1. A method for emotion recognition based on modality generation of electroencephalogram and eye movement signals, characterized in that, Includes the following steps: Step 1, Manual Feature Extraction: Extract differential entropy features from the input EEG signal and preprocess the input eye movement signal; Step 2, extracting EEG spatial features: The extracted differential entropy features are input into the asymmetric spatial layer of the emotion recognition network TSception to extract spatial features. Then, the spatial features are input into the efficient channel attention mechanism ECA to obtain EEG spatial features with channel attention attributes. Step 3, Extracting EEG temporal features: The extracted differential entropy features are input into a bidirectional long short-term memory network integrating a self-attention mechanism. To further extract temporal dependency features, we obtain attention-enhanced temporal features; and through a fully connected layer, we integrate and transform the temporal features to obtain EEG temporal features. Step 4, extract eye movement spatiotemporal features: extract features from the preprocessed eye movement signal to obtain discriminative eye movement spatiotemporal features; Step 5, Feature splicing and modal alignment: The EEG spatial features and EEG temporal features are spliced ​​to obtain EEG spatiotemporal features. Modal alignment loss based on angular cosine distance is introduced as a regularization term to promote the consistency of cross-modal semantic representation of EEG signals and eye movement signals. Step 6, Modal Fusion: The cross-modal information fusion module CMEF is used to fuse the spatiotemporal features of EEG and eye movement to obtain fused features; Step 7, Pre-training: Perform cross-modal generative pre-training to obtain the pre-trained model; Step 8, Model Testing: Supervised fine-tuning of the pre-trained model yields the emotion recognition model, achieving the final emotion classification.

2. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, In step 1, the process of extracting differential entropy features from the EEG signal is as follows: First, a set of bandpass filters is used to decompose the EEG signal into several physiological frequency bands; Then, the time-domain signal of each physiological frequency band is subjected to fast Fourier transform to obtain the power spectrum of the physiological frequency band. Finally, the differential entropy characteristics are calculated from the power spectrum of each physiological frequency band. Assuming that the EEG signal follows a Gaussian distribution, the differential entropy characteristics of the EEG signal are calculated as follows: ； in, This indicates the input brainwave signal; This represents the average value of the signal over a time interval; Indicates standard deviation; Represents the characteristics of differential entropy; Represents pi; Represents an exponential function; Represented by constants A logarithmic function with base 0; The process of preprocessing eye movement signals is as follows: For eye-tracking signals, principal component analysis is used to reduce dimensionality and remove correlations between features. The process is as follows: ； in, The input eye movement signal; It is the square root of the inverse of the diagonal covariance matrix; The projection matrix formed by the selected principal component directions; This represents the preprocessed eye movement signal.

3. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, In step 2, the asymmetric spatial layer of the emotion recognition network TSception includes a global convolutional kernel and a hemispherical convolutional kernel. The global convolutional kernel extracts spatial dependency features between whole-brain channels to obtain global spatial features. The hemispherical convolution kernel models the correspondence and differences between the left and right hemispheres, obtaining hemispherical spatial features. Global spatial features and hemispherical spatial characteristics Spatial features are obtained by concatenating them along the feature dimension after average pooling. The specific calculation process is as follows: ； ； ； in, Represents the global convolutional kernel weights. Represents a one-dimensional convolution kernel; For global spatial features; Indicates the weights of the hemispherical convolution kernel; It is a hemispherical spatial feature; For activation functions; Indicates the concatenation function; This indicates an average pooling operation; express function; Spatial features; This indicates a batch normalization operation; Represents the characteristics of differential entropy; The obtained spatial features The input efficient channel attention mechanism, ECA, first aggregates the multi-channel feature information obtained through global average pooling, then generates channel weights through one-dimensional convolution, and finally uses... The function generates the weight ratio for each channel, and the ECA input is multiplied by the weight ratio to obtain the EEG spatial features with channel attention attributes. The specific process is as follows: The obtained spatial features The efficient channel attention mechanism ECA first aggregates the multi-channel feature information obtained through global average pooling, as follows: ； in, , , They are respectively Total height, total width, total number of channels , , They are respectively Height index, width index, channel index; For the first The output values ​​of each channel after global average pooling; Next, the kernel size is... The one-dimensional convolution generates channel weights as follows: ； in, This is the feature vector output by global average pooling. , For the first The output values ​​of each channel after global average pooling The kernel size represents the convolution kernel size. One-dimensional convolution operation, This represents the intermediate weight vector of the one-dimensional convolution output. The calculation formula is: ； in, Indicates the kernel size; Regarding the total number of channels The function; express The logarithm of ; Represents the offset of the linear mapping; Indicates closest Odd numbers; This is the scaling factor; Last use The function will Mapping to the [0,1] interval, we obtain the normalized weights for each channel, and then the spatial features are... Multiplying by the channel weights yields EEG spatial features with channel attention attributes. The specific process is as follows: ； ； in, for Activation function; This is the intermediate weight vector output by a one-dimensional convolution; This is the normalized channel weight vector; This represents channel-by-channel multiplication, i.e. The channel equals Channels and weights Multiply; This refers to the spatial characteristics of brain electrical activity.

4. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, In step 3, the extracted differential entropy features Input-integrated bidirectional long short-term memory network with self-attention mechanism The temporal features after attention enhancement were obtained. Then, a fully connected layer is used to process the temporal features. By integrating and transforming dimensions, the temporal characteristics of the EEG were obtained. The specific calculation process is as follows: ； ； ； in, It is a feature of differential entropy; This represents a bidirectional long short-term memory network; for The output sequence of hidden states; for transpose; Indicates temporal characteristics; To query the transformation matrix; for Transpose of; The key transformation matrix is... Value transformation matrix Transpose of; The dimension of the key vector; for Activation function; Indicates a fully connected layer; This refers to the temporal characteristics of electroencephalogram (EEG).

5. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, In step 4, firstly, a multi-scale one-dimensional temporal convolution kernel is designed for eye movement signals to effectively capture the changing trend of pupil diameter, continuous fixation events, and instantaneous salivation and blinking behaviors. Subsequently, global convolution kernels and hemispherical convolution kernels were introduced to spatially model the binocular channels, respectively characterizing the overall correlation of eye movement signals and the spatial asymmetry between the left and right eyes. By adapting the temporal-spatial modeling method to eye movement signals, discriminative spatiotemporal features of eye movements are ultimately obtained. The specific calculation process is as follows: ； ； ； in, Indicates the first A one-dimensional temporal convolution kernel; This represents the number of one-dimensional temporal convolution kernels. This represents the preprocessed eye movement signal; For activation functions; For the first The output of a one-dimensional time convolution kernel; A temporal feature matrix is ​​formed by concatenating the outputs of one-dimensional temporal convolution kernels; For splicing operations; Indicates a fully connected layer; Represents the co-convolution kernel; This represents the differential convolution kernel.

6. The method for emotion recognition based on EEG and eye movement signals according to claim 1, characterized in that, The specific process of step 5 is as follows: First, spatial characteristics of brainwaves With EEG temporal characteristics By splicing the data, the spatiotemporal features of the brainwaves were obtained. ; ； in, For splicing operations; Then, each is passed through a learnable first linear mapping layer. Second linear mapping layer Spatiotemporal characteristics of EEG Spatiotemporal characteristics of eye movements Projected onto a shared semantic space of the same dimension: ； in, This is the semantic vector of the EEG signal; This is the semantic vector of the eye-tracking signal; This is the first linear mapping layer; This is the second linear mapping layer; Next, the angular cosine distance is defined as follows: ； in, for and Cosine similarity between them; It is the inverse cosine function; Represents pi; for and The angular cosine distance between them; The modal alignment loss is the expected value of the angular cosine distance: ； in, This is the modal alignment loss; This is the expected value; Represents a pair of spatiotemporal features of brainwaves and eye movement spatiotemporal features .

7. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, The specific process of step 6 is as follows: First, the aligned spatiotemporal features of the brainwaves With eye movement spatiotemporal features Projected onto the same dimension through independent linear layers: ； ； in, Linear operations on the spatiotemporal characteristics of brainwaves; The projected EEG characteristics; Linear operations on eye-tracking spatiotemporal features; The eye movement features after projection; Then, and As the input to the cross-modal information fusion module CMEF, the core of CMEF is bidirectional cross-attention; Finally, the outputs of the bidirectional cross-attention are fused, then normalized and pooled to obtain the fused features via a feedforward neural network. The process is as follows: ； in, Features of fusion; Indicates pooling operation; This indicates a normalization operation; It is a feedforward neural network; This refers to the brainwave's output of attention in response to eye movements; This refers to the attention output of eye movements to brainwaves.

8. The emotion recognition method based on modality generation of EEG and eye movement signals according to claim 7, characterized in that, In step 6, the specific execution process of bidirectional cross-attention is as follows: The projected EEG features serve as a query, while the projected eye movement features provide information: ； in, This refers to the brainwave's output of attention in response to eye movements; for Activation function; To query the weight matrix; This is the key weight matrix; Value weight matrix; The dimension of the key vector; This is the transpose of the matrix; The projected EEG characteristics; The eye movement features after projection; The projected eye movement features serve as the query, while the projected EEG features provide information: ； in, This refers to the attention output of eye movements to brainwaves.

9. The method for emotion recognition based on EEG and eye movement signals according to claim 8, characterized in that, In step 7, a modality generation task is designed in the pre-training stage. The modality generation task generates a label sequence of another modality by using a certain modality as a condition and through an autoregressive method. The autoregressive approach employs an autoregressive generative framework, specifically, a fusion representation of classification labels. As an initial context representation, a lightweight head-by-head label-based EEG sequence is generated to predict EEG sequences; let the length of the EEG label sequence to be generated be... The generation process is performed recursively: ； ； in, A fusion representation for classification labels; This serves as the initial context representation; For linear generation of the head; For the generated first One EEG marker; This is the hidden state from the previous step; The length of the EEG marker sequence to be generated; The objective function of the modality generation task is the modality generation loss: ； in, Represents the modal generation loss; Indicates the first One sample; The number of samples; It is the first prediction based on eye movement signals. In the nth sample One EEG marker, It is the first In the nth sample One real EEG marker; Modality alignment loss is introduced as an auxiliary supervision during the pre-training phase. With mode generation loss Together they constitute the pre-training objective: ； in, The overall goal of pre-training; This is the modal alignment loss; These are the weighting coefficients for the modal alignment loss.

10. The method for emotion recognition based on modality generation of EEG and eye movement signals according to claim 1, characterized in that, In step 8, supervised fine-tuning is performed under both subject-dependent and subject-independent experimental protocols. In the subject-dependent experiment, a five-fold cross-validation strategy was used and executed independently within each subject. Training and test data were obtained from the same subject, and the mean accuracy and standard deviation of all subjects were calculated. In the independent experiment, leave-one-subject cross-validation was used. Each time, all data from one subject were left as the test set, and the data from all other subjects were used as the training set for fine-tuning and evaluation. After going through all subjects, the mean and standard deviation of the overall test results were used as the final performance index.

Images