Electroencephalogram self-supervised emotion recognition system based on time sequence prediction and spatial reconstruction

Abstract

An electroencephalogram self-supervised emotion recognition system based on time sequence prediction and spatial reconstruction comprises an encoder module, a time sequence prediction module, a spatial feature extraction module and a symmetric decoder module. Extracting preliminary spatio-temporal features through a basic neural network structure such as time sequence convolution; the time sequence prediction module carries out future time step prediction on the preliminary spatial-temporal characteristics; the spatial feature extraction module carries out CNN-Transform structure-based spatial dimension attention feature processing according to the initial spatial-temporal features extracted by the encoder module to obtain spatial representation used for reconstructing a randomly shielded channel, and then the symmetric decoder module carries out spatial dimension reconstruction on the spatial representation. According to the method, under the condition that manual features are not needed, universal spatio-temporal representation can be learned from original EEG signals, and end-to-end feature modeling is achieved; by effectively fusing mask reconstruction and context contrast learning, the joint perception ability of the model to local and global features is improved, and the discrimination of representation and the migration generalization ability are enhanced.

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural network applications, specifically an EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction. Background Art

[0002] With the widespread application of electroencephalogram (EEG) signals in fields such as emotion recognition, emotion recognition models based on deep learning have developed rapidly. However, most existing EEG emotion recognition technologies are unable to directly decode emotions "end-to-end" from raw signals. Summary of the Invention

[0003] In response to the existing technology, the present invention uses a spectrum-space-time multi-view mask reconstruction mechanism to achieve self-supervised feature learning, but its input still relies on manual features such as artificially constructed spectrograms, failing to achieve end-to-end learning from the original EEG signals, easily causing the loss of temporal information, limiting the model's ability to model the original dynamic structure of EEG, and lacking a joint design with contrastive learning, making it difficult to enhance the discriminability and robustness of sample representation, reducing its performance in migration scenarios. An EEG self-supervised emotion recognition system based on temporal prediction and spatial reconstruction is proposed, which can learn universal spatiotemporal representations from original EEG signals without the need for manual features, thereby achieving end-to-end feature modeling; by effectively integrating mask reconstruction and contextual contrastive learning, the model's joint perception of local and global features is improved, thereby enhancing the discriminability and migration generalization capabilities of the representation.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction, comprising: an encoder module, a time series prediction module, a spatial feature extraction module and a symmetric decoder module, wherein: the encoder module extracts preliminary spatiotemporal features based on multi-channel EEG signals received in the form of time series through basic neural network structures such as time series convolution; the time series prediction module predicts future time steps for the preliminary spatiotemporal features; the spatial feature extraction module performs spatial dimension attention feature processing based on a CNN-Transformer structure on the preliminary spatiotemporal features extracted by the encoder module to obtain a spatial representation for reconstructing randomly masked channels, and the symmetric decoder module then reconstructs the spatial dimension of this spatial representation.

[0006] The present invention relates to a self-supervised EEG representation learning method based on the above system. In the pre-training stage, a universal encoder with spatiotemporal perception capabilities is trained using large-scale unlabeled EEG data. In the fine-tuning stage, the network parameters of the universal encoder are frozen, and a linear classifier network can be trained using a small number of labeled target task training sets to achieve recognition of emotional states.

[0007] The pre-training phase includes the time series prediction task, spatial reconstruction task, and contextual contrast learning mechanism, which fully exploits the spatiotemporal features of the original EEG signals and implements end-to-end feature representation learning. Specifically, it includes:

[0008] Step 1: Randomly apply data augmentation processing to the original EEG signal, including random jitter, scale scaling and segment rearrangement, to obtain two sets of enhanced data, which are input into the encoder module together with the untransformed original signal to extract preliminary spatiotemporal features;

[0009] Step 2: The preliminary spatiotemporal features corresponding to the augmented data are input into the time series prediction module based on the Transformer structure. Based on the randomly selected moment features, the feature representation of the other augmented data at several future time steps is predicted, and the context contrast loss is used to enhance the consistency of features within the same sample.

[0010] Step 3: Input the preliminary spatiotemporal features corresponding to the original signal into the spatial feature extraction module, randomly mask the features of some channels, reconstruct the spatial features of the masked channels through the symmetric decoder module, and optimize the network parameters through the spatial reconstruction loss function.

[0011] The fine-tuning stage specifically includes:

[0012] Step a: Freeze the network parameters of the general encoder module obtained in the pre-training phase and perform feature extraction using the labeled target task dataset;

[0013] Step b: Input the extracted EEG signal features into the linear classifier network and fine-tune the network parameters using the cross-entropy loss function;

[0014] Step c: Use the validation set data to select the optimal model parameters and perform performance verification on the test set to achieve the EEG emotion classification task.

[0015] Technical Effects

[0016] By integrating temporal prediction tasks, spatial reconstruction tasks, and contextual comparative learning, this method can directly operate on a variety of raw EEG datasets and fully exploit their inherent spatiotemporal characteristics, thereby constructing a universal and generalizable EEG encoder network. Compared with existing technologies, this method significantly improves the accuracy of EEG emotion recognition tasks with minimal labeled data, achieving a relative improvement of over 20% in emotion recognition accuracy compared to existing supervised learning methods. Furthermore, the cross-dataset pre-training and fine-tuning results demonstrate the method's powerful generalization and transfer capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0018] like Figure 1 As shown, this embodiment relates to an EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction, including: an encoder module, a time series prediction module, a spatial feature extraction module and a symmetric decoder module, wherein: the encoder module extracts preliminary spatiotemporal features based on the received multi-channel EEG signals in the form of time series through basic neural network structures such as time series convolution; the time series prediction module predicts the future time steps of the preliminary spatiotemporal features; the spatial feature extraction module performs spatial dimension attention feature processing based on the CNN-Transformer structure based on the preliminary spatiotemporal features extracted by the encoder module to obtain a spatial representation for reconstructing randomly masked channels, and the symmetric decoder module reconstructs the spatial dimension of this spatial representation to obtain a complete EEG spatial feature representation result.

[0019] The encoder module maintains parameter sharing throughout the training phase, both for self-supervised training and for feature extraction in the subsequent fine-tuning phase.

[0020] The encoder module includes: a CNN convolution unit, a batch normalization unit, a nonlinear activation unit and a pooling unit, wherein: the CNN convolution unit performs convolution operation processing based on the input original EEG signal sequence information to obtain a preliminary representation of channel-independent features; the batch normalization unit performs batch normalization processing based on the preliminary representation information output by the CNN convolution unit to obtain a stable feature representation result; the nonlinear activation unit performs nonlinear activation function processing based on the standardized feature information output by the batch normalization unit to obtain a feature result with nonlinear expression capability; the pooling unit performs pooling processing based on the feature information output by the nonlinear activation unit to obtain a preliminary spatiotemporal feature representation of the final dimensionality reduction.

[0021] The time series prediction module includes: a time series Transformer unit, a position embedding unit, a prediction token unit, a projection mapping unit and a context comparison unit, wherein: the time series Transformer unit performs feature fusion processing of the self-attention mechanism according to the spatiotemporal feature representation information output by the encoder module to obtain an enhanced time series feature representation result; the position embedding unit performs sine and cosine position encoding processing according to the information of the moment to be predicted to obtain a time series feature representation result with position information; the prediction token unit adds a learnable special token according to the feature representation information after position embedding to obtain context features for future time series prediction; the projection mapping unit performs linear mapping processing according to the above-mentioned predicted context feature information to obtain feature prediction results for future moments; the context comparison unit performs context comparison learning processing according to the prediction result information after projection to obtain a more discriminative time series feature representation result.

[0022] The spatial feature extraction module includes: a spatial position encoding unit, a spatial Transformer encoding unit, a feature mask unit and a spatial feature fusion unit, wherein: the spatial position encoding unit adds sine and cosine position encoding processing according to the spatial feature information after channel dimension division to obtain features with spatial position relationship; the spatial Transformer encoding unit performs spatial attention mechanism processing according to the spatial feature information after position encoding to obtain enhanced spatial feature representation; the feature mask unit randomly selects and masks part of the channel features according to the enhanced spatial feature representation information to obtain the spatial feature representation to be reconstructed; the spatial feature fusion unit performs feature fusion processing according to the unmasked spatial feature representation information to obtain a complete input representation for the reconstruction task.

[0023] The symmetric decoder module includes: a spatial Transformer decoding unit, a feature reconstruction unit and a feature recovery unit, wherein: the spatial Transformer decoding unit performs attention mechanism processing based on the initial mask features and the fused spatial feature information to obtain the decoded feature representation; the feature reconstruction unit performs channel-level feature reconstruction processing based on the decoded feature representation information to obtain the spatial feature representation that restores the channel integrity; the feature recovery unit performs channel dimension merging processing based on the reconstructed spatial feature representation information to obtain a reconstructed feature output that matches the initial input.

[0024] After specific actual experiments, we conducted experiments in the specific environment settings of three public emotional EEG datasets, SEED, SEED-IV, and SEED-V, with the following parameters: the pre-training learning rate is 3×10⁻ 4 , a batch size of 256; during the fine-tuning phase, the learning rate was 1×10⁻³, the batch size varied from 16 to 256, the prediction step was 12, and the masking ratio was 50%; the model encoder used a 1D-CNN network structure with a kernel size of 8. The experimental data results are shown in Table 1.

[0025] Table 1

[0026] Compared to existing technologies, this invention achieves end-to-end modeling of raw EEG signals: It extracts spatiotemporal features directly from raw multi-channel EEG signals without relying on artificially constructed spectrograms or other handcrafted features, improving the completeness and generalization of feature extraction. This advantage is due to the use of a unified encoder structure and pre-training combined with time series prediction and spatial reconstruction. 2. Fusion of reconstruction and contrast mechanisms to enhance representation discriminability: Unlike existing technologies that rely solely on reconstruction or a single self-supervised task, this technical solution combines mask reconstruction with contextual contrastive learning to enhance inter-sample discrimination while performing local reconstruction, thereby improving the accuracy of emotion classification with low-labeled data. 3. Excellent scalability to adapt to the trend of large-scale models: This invention adopts an end-to-end learning framework, eliminating reliance on artificial features. This makes the model more suitable for integration with currently popular high-capacity neural networks (such as Transformer-like architectures), and offers good structural scalability and parameter scalability. This design provides a more convenient path for the deployment and migration of large models for EEG emotion recognition, driving the development of models towards higher performance and greater versatility.

[0027] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. An EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction, characterized by: include: The encoder module, time series prediction module, spatial feature extraction module and symmetric decoder module are as follows: the encoder module extracts preliminary spatiotemporal features based on the received multi-channel EEG signals in the form of time series through basic neural network structures such as time series convolution; the time series prediction module predicts the future time steps of the preliminary spatiotemporal features; The spatial feature extraction module performs spatial dimension attention feature processing based on the CNN-Transformer structure according to the preliminary spatiotemporal features extracted by the encoder module to obtain a spatial representation for reconstructing the randomly masked channels. The symmetric decoder module reconstructs the spatial dimension of this spatial representation to obtain a complete EEG spatial feature representation result.

2. The EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction according to claim 1 is characterized in that: The encoder module used for feature extraction in self-supervised training and subsequent fine-tuning stages includes: CNN convolution unit, batch normalization unit, nonlinear activation unit and pooling unit, among which: the CNN convolution unit performs convolution operation processing based on the input original EEG signal sequence information to obtain a preliminary representation of channel-independent features; the batch normalization unit performs batch normalization processing based on the preliminary representation information output by the CNN convolution unit to obtain a stable feature representation result; the nonlinear activation unit performs nonlinear activation function processing based on the standardized feature information output by the batch normalization unit to obtain a feature result with nonlinear expression ability; the pooling unit performs pooling processing based on the feature information output by the nonlinear activation unit to obtain the final dimensionality reduction preliminary spatiotemporal feature representation.

3. The EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction according to claim 1 is characterized in that: The time series prediction module includes: a time series Transformer unit, a position embedding unit, a prediction token unit, a projection mapping unit and a context comparison unit, wherein: the time series Transformer unit performs feature fusion processing of the self-attention mechanism according to the spatiotemporal feature representation information output by the encoder module to obtain an enhanced time series feature representation result; the position embedding unit performs sine and cosine position encoding processing according to the information of the moment to be predicted to obtain a time series feature representation result with position information; the prediction token unit adds a learnable special token according to the feature representation information after position embedding to obtain context features for future time series prediction; the projection mapping unit performs linear mapping processing according to the above-mentioned predicted context feature information to obtain feature prediction results for future moments; the context comparison unit performs context comparison learning processing according to the prediction result information after projection to obtain a more discriminative time series feature representation result.

4. The EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction according to claim 1 is characterized in that: The spatial feature extraction module includes: a spatial position encoding unit, a spatial Transformer encoding unit, a feature mask unit and a spatial feature fusion unit, wherein: the spatial position encoding unit adds sine and cosine position encoding processing according to the spatial feature information after channel dimension division to obtain features with spatial position relationship; the spatial Transformer encoding unit performs spatial attention mechanism processing according to the spatial feature information after position encoding to obtain enhanced spatial feature representation; the feature mask unit randomly selects and masks part of the channel features according to the enhanced spatial feature representation information to obtain the spatial feature representation to be reconstructed; the spatial feature fusion unit performs feature fusion processing according to the unmasked spatial feature representation information to obtain a complete input representation for the reconstruction task.

5. The EEG self-supervised emotion recognition system based on time series prediction and spatial reconstruction according to claim 1 is characterized in that: The symmetric decoder module includes: a spatial Transformer decoding unit, a feature reconstruction unit and a feature recovery unit, wherein: the spatial Transformer decoding unit performs attention mechanism processing based on the initial mask features and the fused spatial feature information to obtain the decoded feature representation; the feature reconstruction unit performs channel-level feature reconstruction processing based on the decoded feature representation information to obtain the spatial feature representation that restores the channel integrity; the feature recovery unit performs channel dimension merging processing based on the reconstructed spatial feature representation information to obtain a reconstructed feature output that matches the initial input.

6. A self-supervised EEG representation learning method based on the system according to any one of claims 1 to 5, characterized in that: In the pre-training stage, a universal encoder with spatiotemporal perception capabilities is trained using large-scale unlabeled EEG data; in the fine-tuning stage, the network parameters of the universal encoder are frozen, and a linear classifier network can be trained using a small amount of labeled target task training sets to achieve recognition of emotional states.

7. The self-supervised EEG representation learning method according to claim 6, characterized in that: The pre-training stage specifically includes: Step 1: Randomly apply data augmentation processing to the original EEG signal, including random jitter, scale scaling and segment rearrangement, to obtain two sets of enhanced data, which are input into the encoder module together with the untransformed original signal to extract preliminary spatiotemporal features; Step 2: The preliminary spatiotemporal features corresponding to the augmented data are input into the time series prediction module based on the Transformer structure. Based on the randomly selected moment features, the feature representation of the other augmented data at several future time steps is predicted, and the context contrast loss is used to enhance the consistency of features within the same sample. Step 3: Input the preliminary spatiotemporal features corresponding to the original signal into the spatial feature extraction module, randomly mask the features of some channels, reconstruct the spatial feature representation of the masked channels through the symmetric decoder module, and optimize the network parameters through the spatial reconstruction loss function.

8. The self-supervised EEG representation learning method according to claim 6, characterized in that: The fine-tuning stage specifically includes: Step a: Freeze the network parameters of the general encoder module obtained in the pre-training phase and perform feature extraction using the labeled target task dataset; Step b: Input the extracted EEG signal features into the linear classifier network and fine-tune the network parameters using the cross-entropy loss function; Step c: Use the validation set data to select the optimal model parameters and perform performance verification on the test set to achieve the EEG emotion classification task.

Images