Motor imagery recognition method based on three-dimensional electroencephalogram signal representation and Mama

Abstract

The invention relates to the field of brain-computer interfaces, in particular to a motor imagery recognition method based on three-dimensional electroencephalogram signal representation and Mama, which comprises the following steps: collecting electroencephalogram data, storing the electroencephalogram data in a two-dimensional matrix form according to electrode positions, intercepting the electroencephalogram data according to a set time step length, and stacking the intercepted electroencephalogram data along a time dimension to form three-dimensional electroencephalogram data; in the first Mama module, processing each time slice of the three-dimensional electroencephalogram data in sequence, and stacking the three-dimensional electroencephalogram data along the time dimension at the outlet end to obtain first three-dimensional feature data; respectively segmenting the first three-dimensional feature data according to rows and columns in the form of a two-dimensional matrix, respectively inputting the first three-dimensional feature data into a second Mama module and a third Mama module, adding the output of the second Mama module and the output of the third Mama module to obtain a fused feature, and performing classification by using the feature; compared with the prior art, the method has the advantages that efficient and low-calculation-cost motor imagery classification of the EEG data is realized.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface (BCI), and in particular to a motor imagery recognition method based on three-dimensional electroencephalogram signal representation and Mamba. Background Art

[0002] Brain-computer interface (BCI) is a technology that uses EEG signals to communicate with external devices. In recent years, it has attracted widespread attention in the fields of medical rehabilitation, intelligent control, and human-computer interaction. In particular, in the motor imagery (MI) classification task, users control external devices such as prostheses or robots by imagining limb movements. However, traditional EEG signal processing and classification methods have certain limitations in dealing with complex spatiotemporal features and coping with individual differences.

[0003] At present, most common motor imagery EEG signal classification methods are based on two-dimensional feature extraction, such as using time series analysis, frequency domain features or simple spatial projection. However, such methods fail to fully utilize the three-dimensional characteristics of EEG signals (space, electrode distribution and time), making it difficult for the model to capture the rich spatiotemporal information of the signal. In addition, conventional feature extraction methods are inadequate in dealing with nonlinear and non-stationary signals and are easily affected by noise and artifacts, thus affecting the classification performance and the robustness of the model. Summary of the invention

[0004] In order to solve the shortcomings of the prior art in the classification of motor imagery EEG signals, the present invention proposes a motor imagery recognition method based on three-dimensional EEG signal representation and Mamba, which specifically comprises the following steps:

[0005] Collecting EEG data, preprocessing the EEG data, and saving the preprocessed EEG data in the form of a two-dimensional matrix according to the electrode positions;

[0006] Set the time step, intercept the EEG data in the form of a two-dimensional matrix according to the set time step, and stack the intercepted data along the time dimension to form three-dimensional EEG data;

[0007] Construct a feature extraction module consisting of three Mamba modules. In the first Mamba module, each time segment of the three-dimensional EEG data is processed in turn, and stacked along the time dimension at the source end to obtain the first three-dimensional feature data;

[0008] The first three-dimensional feature data is divided into rows and columns in the form of a two-dimensional matrix;

[0009] The row slices and column slices obtained by segmentation are input into the second Mamba module and the third Mamba module respectively, and the outputs of the second Mamba module and the third Mamba module are added together to obtain the fusion feature;

[0010] The obtained fusion features are input into the pre-trained classifier to obtain the probability distribution of motor imagery categories.

[0011] Furthermore, the EEG data are preprocessed including downsampling, bandpass filtering and artifact removal.

[0012] Furthermore, the preprocessed EEG data are saved in the form of a two-dimensional matrix according to the electrode positions, that is, the preprocessed EEG data are mapped into a two-dimensional matrix that conforms to the international 10-20 system electrode distribution.

[0013] Furthermore, the 3D EEG data is represented as Where C represents the number of electrodes, T represents the number of samples at a time point, and F represents the length of data collected by a single electrode in one time step.

[0014] Furthermore, the classifier consists of two fully connected layers, a SELU activation layer and a Softmax layer.

[0015] The present invention combines 3D EEG representation and Mamba feature extraction module to comprehensively and effectively extract spatiotemporal features from EEG signals, significantly improving the classification accuracy of motor imagery tasks. Compared with traditional methods, the present invention can reduce information loss while maintaining model stability, enhance generalization capabilities for different users and tasks, and is suitable for real-time brain-computer interface applications such as prosthetic control, rehabilitation training, and other intelligent interactive systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The training flow chart of the EEG signal motion recognition model of the present invention;

[0017] Figure 2 Figure 2 shows the cross-subject classification accuracy of different models on the BCI competition IV 2a dataset. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In order to solve the shortcomings of the prior art in the classification of motor imagery EEG signals, the present invention proposes a motor imagery recognition method based on three-dimensional EEG signal representation and Mamba, such as Figure 1 , the specific steps are as follows:

[0020] Collecting EEG data, preprocessing the EEG data, and saving the preprocessed EEG data in the form of a two-dimensional matrix according to the electrode positions;

[0021] Set the time step, intercept the EEG data in the form of a two-dimensional matrix according to the set time step, and stack the intercepted data along the time dimension to form three-dimensional EEG data;

[0022] Construct a feature extraction module consisting of three Mamba modules. In the first Mamba module, each time segment of the three-dimensional EEG data is processed in turn, and stacked along the time dimension at the source end to obtain the first three-dimensional feature data;

[0023] The first three-dimensional feature data is divided into rows and columns in the form of a two-dimensional matrix;

[0024] The row slices and column slices obtained by segmentation are input into the second Mamba module and the third Mamba module respectively, and the outputs of the second Mamba module and the third Mamba module are added together to obtain the fusion feature;

[0025] The obtained fusion features are input into the pre-trained classifier to obtain the probability distribution of motor imagery categories.

[0026] like Figure 1 As shown, the present invention proposes a motor imagery EEG signal classification method based on 3D EEG representation and Mamba feature extraction, comprising the following steps:

[0027] S1. Collect the EEG signals of users when they perform different motor imagery tasks, and pre-process them through downsampling, bandpass filtering, and ICA artifact removal to remove noise and artifacts. ;

[0028] S2, mapping the preprocessed EEG data into a two-dimensional electrode matrix that conforms to the international 10-20 system, and stacking it along the time dimension to form a three-dimensional EEG tensor X;

[0029] S3, input the three-dimensional EEG tensor X into the Mamba feature extraction module, extract spatiotemporal features through the forward and masked backward SSM layers, and obtain the output matrix;

[0030] S4, input the extracted features into the classifier, which includes a fully connected layer, a SELU activation layer and a Softmax layer, and finally outputs the classification result of the motor imagery task;

[0031] S5. Calculate the overall loss function in model training, including classification loss, forward stability loss, and backward consistency loss, and use the ADAM optimizer for training and parameter update;

[0032] The present invention uses the 2a subtask data set of BCI Competition IV as experimental data. The data set contains EEG signals of 9 subjects. Each subject conducted two experiments, each experiment included 6 rounds, each round contained 48 trials, and each motor imagery task (left hand, right hand, feet and tongue) was performed 12 times, totaling 288 trials per experiment. During the experiment, the subjects were asked to sit in a comfortable chair and perform motor imagery tasks according to the prompts in front of the computer screen. At the beginning of each trial, a fixed cross was displayed on the screen accompanied by a prompt sound. After 2 seconds, an indicator arrow appeared and remained for 1.25 seconds, prompting the subject to perform the corresponding motor imagery task until the prompt ended at 6 seconds.

[0033] Preprocess the EEG signals, including:

[0034] First, the sampling rate was reduced from 250 Hz to 200 Hz to reduce computational complexity and data redundancy while retaining sufficient frequency information for subsequent analysis. Then, the signal was filtered using a 0.5-50 Hz bandpass filter to remove low-frequency drift and high-frequency noise, thereby ensuring that the signal mainly contained useful information in the frequency band related to motor imagery. Subsequently, independent component analysis (ICA) was applied to remove artifacts from the preprocessed EEG signal, especially non-EEG artifacts such as electrooculogram (EOG) and electromyography, to improve signal purity and analysis accuracy.

[0035] After data preprocessing, the EEG data are segmented into non-overlapping time windows of 1 second in length to generate uniform time segments. These time segments contain the EEG signals of each electrode within the predetermined time window. The data within each time window will be reorganized into a two-dimensional matrix that conforms to the international 10-20 system of electrode positions. The matrix encodes the spatial position relationship of the electrodes, allowing subsequent feature extraction to simultaneously consider spatial correlation.

[0036] The two-dimensional electrode matrix of each time window is stacked along the time dimension to form a three-dimensional EEG tensor Where C represents the number of electrode channels, T represents the number of time steps, and F represents the feature dimension calculated at each electrode position. This three-dimensional structure construction method ensures the spatial and temporal integrity of EEG data, and can provide information input including the spatial layout of electrodes and the dynamic changes of time series to the subsequent feature extraction module.

[0037] The core purpose of this 3D EEG data construction process is to transform the original EEG signal into a multi-dimensional structure, provide multi-angle feature expression, and optimize the performance of downstream tasks such as motor imagery classification. Through this preprocessing and 3D data construction, the useful information of the EEG signal is refined and organized, providing high-quality input data for the subsequent feature extraction module.

[0038] During the training process, the dataset was divided using Leave-One-Subject-Out (LOSO) cross-validation. In each validation, the data of one subject was selected as the test set, and the data of the other 8 subjects were used as the training set and validation set. This method ensures that the model does not see the data of the test subject during training, so that the generalization performance of the model on completely unknown subjects can be evaluated. By using each subject as the test set in turn, the overall performance of the model can be evaluated by the average results of multiple trainings.

[0039] The preprocessed data is input into the Mamba feature extraction module to fully extract spatial and temporal features. The Mamba module is a bidirectional structure consisting of a forward SSM layer and a masked backward SSM layer, which is designed to capture the spatiotemporal dependencies and sequence features in EEG signals.

[0040] The forward SSM layer is responsible for processing the time-step features of the input tensor. Specifically, the input tensor X is first transformed linearly to generate query, key, and value matrices. These matrices are used to calculate the attention weight matrix S, thereby highlighting the features that are important at different time steps. This process helps the model capture the correlation and change rules of the EEG signal in the time dimension, enabling it to focus on the signal segments that are critical to the classification task at different time steps.

[0041] The masked backward SSM layer further aggregates time series information by updating and calculating the recursive state matrix. This masking process allows the model to only access the current and previous steps at each time step, forming a feature capture mechanism similar to causality. This backward mechanism effectively enhances the consistency and stability of the model in the time series, making the extracted features more temporally relevant.

[0042] By combining the forward and backward SSM layers, the Mamba module can achieve the fusion and enhancement of bidirectional information, thereby extracting richer spatiotemporal features. The output feature matrix contains the high-dimensional feature representation obtained after bidirectional feature extraction, which retains the complex dynamic characteristics and spatial dependencies of the EEG signal.

[0043] In the Mamba module, the forward processing uses the linear layer, SSM layer, nonlinear activation layer and residual fusion unit to process the data in sequence, including:

[0044] x′ t =Wx t +b

[0045] h t =Ah (t-1) +Bx′ t

[0046] y t =Ch t +Dx′ t

[0047] y′ t =ReLU(y t )

[0048] y out =y′ t +x′ t

[0049] Among them, x t represents the data input to the Mamba module at time step t, x′ t represents the features obtained after the data input to the Mamba module at time step t is mapped by the linear layer; W and b are the weight and bias terms of the linear layer; h t is the hidden state obtained in the forward SSM layer at time step t, t is an integer greater than 0, h0 is initialized to x1, that is, h0 is initialized to the data input into the network at the first time step; h (t-1) represents the hidden state obtained in the forward SSM layer at time step t-1; y t is the feature representation of the forward SSM layer output at time step t; A, B, C, D are learnable state matrices; y′ t represents the output of the nonlinear activation function; y out Represents the output of the residual fusion unit.

[0050] In order to further improve the stability and generalization ability of the model during training, the forward and backward SSM layers apply regularization strategies in the feature extraction process. This strategy ensures the stability of output features between different data batches by suppressing feature fluctuations and maintaining sequence consistency, reducing classification performance fluctuations caused by individual differences or external interference. The high-dimensional feature matrix output by the feature extraction module will be used for downstream classifiers for processing. Through this feature extraction process, the hidden spatiotemporal patterns in the EEG signal are effectively captured and represented, which provides strong support for the accurate classification of the motor imagery task.

[0051] The high-dimensional spatiotemporal feature matrix output by the Mamba feature extraction module is further input into the classifier for processing and classification. The classifier is designed to map complex spatiotemporal features to motor imagery task categories and output the probability distribution of each category. The classifier mainly consists of two fully connected layers, a SELU activation layer, and a Softmax layer to achieve efficient feature mapping and classification.

[0052] The first fully connected layer receives the features output from the Mamba module and performs dimensionality reduction. This step compresses the input feature space to a smaller dimension through linear transformation of the weight matrix to improve the efficiency of subsequent calculations and provide the classifier with a feature combination suitable for the classification task. The reduced-dimensional features are passed to the activation layer for nonlinear mapping.

[0053] The SELU activation layer is applied to the output of the first fully connected layer. The SELU (Scaled Exponential Linear Unit) activation function has a self-regularization property, which can effectively avoid gradient explosion and gradient vanishing problems in deep neural networks, thereby ensuring that the model still has stable gradient flow in the deep structure. This feature is particularly important for the complexity and dynamics of features in EEG signals, ensuring that the features extracted in the motor imagery task can be fully expressed.

[0054] The second fully connected layer is used to further reduce the dimension of the features after SELU activation and combine them into the final classification input features. This process helps to perform the final transformation of the features before the output layer to better adapt to the input requirements of the Softmax layer. After this step, the features are adjusted to a shape and space suitable for classification.

[0055] The Softmax layer is the last layer of the classifier, which is used to map the output of the previous layer into the probability distribution of each motor imagery task. The Softmax function converts the input features into a probability vector, in which each component corresponds to a specific motor imagery category. The sum of the output probabilities is 1, and the model predicts the category with the highest probability as the final motor imagery task classification result. This process can intuitively reflect the model's confidence in different categories.

[0056] The output of the Softmax layer can not only be used to predict the category of the motor imagery task, but also provide the probability value of each category as a confidence indicator for the model decision. In this way, adaptive decisions can be made in the real-time brain-computer interface system based on the confidence of the prediction results. In order to improve the performance of the classifier, the cross entropy loss function is used as the optimization target to measure the difference between the classification output and the true label. During the entire training process, the learning rate is dynamically adjusted in combination with the ADAM optimizer to ensure rapid convergence of the model and prevent overfitting. During training, the validation set is used to monitor the performance of the classifier in real time, and the model parameters are adjusted if necessary to further optimize the performance.

[0057] The loss function when training the feature extraction module and classifier is expressed as:

[0058] L total =L c +λ1L residual +λ2L backward

[0059] Among them, L total represents the total loss function; L c represents the classification loss; L residual represents the forward stability loss; L backward represents the backward consistency loss; λ1 is used to control the residual loss term L residual The importance weight of λ2 is used to control the back propagation loss term L backward The importance weight of .

[0060] Forward stability loss L residual It is expressed as:

[0061]

[0062] Where M represents the number of training iterations; δ is the threshold parameter used to control the acceptable error range; x i Represents the true classification label of the i-th batch of training data; Represents the predicted classification label of the i-th batch of training data.

[0063] Backward consistency loss L residual It is expressed as:

[0064]

[0065] Where N is the number of time steps used to calculate the backward consistency loss; h i represents the hidden state obtained in the forward SSM layer at time step i; represents the hidden state obtained in the backward SSM layer at time step i.

[0066] In this embodiment, the updating process of the hidden state in the backward SSM layer includes:

[0067]

[0068] in, represents the hidden state obtained in the backward SSM layer at time step t, represents the hidden state obtained in the backward SSM layer at time step t+1; A′ and B′ are learnable parameters.

[0069] In this embodiment, if the data of the current T time points have been obtained, then N = T-1, and the backward propagation is updated from time step T-1 to time step 1 in sequence. When t = T-1, the initialization x ′ T Represents the features of the input Mamba module data at time step T after being mapped by the linear layer.

[0070] The model training uses a combination of multiple loss functions to optimize different feature learning objectives. The main loss function is the cross entropy loss, which is used to evaluate the match between the classifier output and the true label. Cross entropy loss can quantify the accuracy of model predictions and is a widely used loss function in supervised learning. In addition, in order to improve the stability and consistency of the model on time series data, forward stability loss and backward consistency loss are used. Forward stability loss constrains the output features of the model in adjacent time steps to ensure the smoothness and stability of the feature extraction process. Backward consistency loss ensures the consistency of the model output in the time dimension by recursively calculating the time series, avoiding drift and instability of long time step features.

[0071] In each training cycle, the model will traverse the entire training set and update the parameters according to the loss function after each batch. After each epoch, the validation set is used to evaluate the model performance, and the loss and accuracy on the validation set are used to determine whether the model is overfitting or underfitting. To prevent the model from overfitting the training data in the later stages of training, an early stopping strategy is applied, that is, training is stopped when the performance on the validation set no longer improves.

[0072] During the training process, hyperparameters such as learning rate and weight parameters of loss function are dynamically adjusted according to the performance on the validation set. The adjustment strategy combines grid search and validation set monitoring to ensure that the model can maintain good robustness under different training conditions. In order to further improve the generalization ability of the model, L2 regularization terms are added during the training process to suppress the excessive growth of model parameters and reduce the risk of overfitting caused by complex models. In addition, random dropout technology is applied in the fully connected layer to randomly drop a certain proportion of neurons in each training cycle to improve the model's robustness to noise and generalization ability.

[0073] After training is complete, the model will be evaluated using the test set. The classification performance of the model is comprehensively measured by calculating indicators such as classification accuracy, recall, and F1 score. Throughout the training and validation process, confusion matrices and other visualization tools are used to help analyze the distribution of classification results and identify the strengths and weaknesses of the model in different motor imagery tasks.

[0074] like Figure 2 Compared with the current mainstream motor imagery classification models, including the traditional FBCSP, pure Transformer EEG Transformer, CNN-Transformer that integrates CNN and Transformer, and CNN channel selection method CMO-CNN, the three-dimensional EEG signal representation and Mamba feature extraction method proposed in the present invention achieved the best average accuracy, exceeding the existing methods by nearly 1 percentage point.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A motor imagery recognition method based on three-dimensional EEG signal representation and Mamba, characterized in that: The following are the specific steps: Collecting EEG data, preprocessing the EEG data, and saving the preprocessed EEG data in the form of a two-dimensional matrix according to the electrode positions; Set the time step, intercept the EEG data in the form of a two-dimensional matrix according to the set time step, and stack the intercepted data along the time dimension to form three-dimensional EEG data; Construct a feature extraction module consisting of three Mamba modules. In the first Mamba module, each time segment of the three-dimensional EEG data is processed in turn, and stacked along the time dimension at the source end to obtain the first three-dimensional feature data; The first three-dimensional feature data is divided into rows and columns in the form of a two-dimensional matrix; The row slices and column slices obtained by segmentation are input into the second Mamba module and the third Mamba module respectively, and the outputs of the second Mamba module and the third Mamba module are added together to obtain the fusion feature; The obtained fusion features are input into the pre-trained classifier to obtain the probability distribution of motor imagery categories.

2. A method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 1, characterized in that: Preprocessing of EEG data includes downsampling, bandpass filtering, and artifact removal.

3. The method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 1, characterized in that: The preprocessed EEG data are saved in the form of a two-dimensional matrix according to the electrode positions, that is, the preprocessed EEG data are mapped into a two-dimensional matrix that conforms to the international 10-20 system electrode distribution.

4. The method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 3, characterized in that: Three-dimensional EEG data is represented as X∈R C×T×F , where C represents the number of electrodes, T represents the number of samples at a time point, and F represents the length of data collected by a single electrode in one time step.

5. The method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 1, characterized in that: The Mamba module includes linear layers, SSM layers, nonlinear activation layers, and residual fusion units to process data, including: x′ t =Wx t +b h t =Ah (t-1) +Bx′ t y t =Ch t +Dx′ t and' t =ReLU(and t ) and out =y′ t +x′ t Among them, x t represents the data input to the Mamba module at time step t, x′ t represents the features obtained after the data input to the Mamba module at time step t is mapped by the linear layer; W and b are the weight and bias terms of the linear layer; h t is the hidden state obtained in the forward SSM layer at time step t; h (t-1) represents the hidden state obtained in the forward SSM layer at time step t-1; y t is the feature representation of the forward SSM layer output at time step t; A, B, C, D are learnable state matrices; y′ t represents the output of the nonlinear activation function; y out Represents the output of the residual fusion unit.

6. The method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 1, characterized in that: The classifier consists of two fully connected layers, a SELU activation layer and a Softmax layer.

7. The method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba according to claim 1, characterized in that: The loss function when training the feature extraction module and classifier is expressed as: L total =L c +λ1L residual +λ2L backward Among them, L total represents the total loss function; L c represents the classification loss; L residual represents the forward stability loss; L backward represents the backward consistency loss; λ1 is used to control the residual loss term L residual The importance weight of λ2 is used to control the back propagation loss term L backward The importance weight of .

8. According to the method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba in claim 7, the forward stability loss L residual It is expressed as: in, M represents the number of training iterations; δ is the threshold parameter used to control the acceptable error range; y i Represents the true classification label of the i-th batch of training data; Represents the predicted classification label of the i-th batch of training data.

9. According to the method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba in claim 7, the backward consistency loss L residual It is expressed as: in, N is the number of time steps used to calculate the backward consistency loss; h i represents the hidden state obtained in the forward SSM layer at time step i; represents the hidden state obtained in the backward SSM layer at time step i.

10. According to the method for motor imagery recognition based on three-dimensional EEG signal representation and Mamba in claim 9, the updating process of the hidden state in the backward SSM layer comprises: in, represents the hidden state obtained in the backward SSM layer at time step t, represents the hidden state obtained in the backward SSM layer at time step t+1; A′ and B′ are learnable parameters.

Images