A motor imagery EEG signal classification and recognition method and system based on data enhancement and multimodal feature fusion

Abstract

The present invention discloses a method and system for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion. The present invention performs data enhancement based on time series decomposition and prediction on the original EEG data; performs time series segmentation on the data enhanced data and the original data, and then extracts the time series features after stacking all channel time series blocks in the same time window; extracts the time-frequency image features after time-frequency analysis on the data enhanced data and the original data; splices the time series features and the time-frequency image features and inputs them into a classifier to obtain a motor imagery classification result. The present invention proposes a time series data enhancement technology based on time series decomposition and prediction, which provides more sufficient training samples for the downstream EEG classification model based on deep learning, and improves the generalization ability and classification effect of the model. It also uses time series and time frequency to extract multimodal features, further improving the effect of EEG signal classification and recognition.

Description

Technical Field

[0001] The present invention belongs to the field of signal processing and pattern recognition in brain-computer interface, and is a method and system for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion. Background Art

[0002] Brain-Computer Interface (BCI) is a system that can directly measure brain activity related to user intentions, convert it into control signals, and then control external devices. Motor imagery (MI), as an important experimental paradigm in BCI technology, refers to the process of simulating or imagining a certain muscle movement through neural activity in the brain without actual movement, involving brain activities in multiple aspects such as sensation, motor planning, and cognition. By analyzing MI signals, the imagineer's movement intentions can be accurately classified to achieve control and operation of external devices.

[0003] At present, many studies have proposed the application of deep learning in EEG signal research, but it is relatively rare compared with the fields of images, videos and texts. Convolutional neural networks (CNN) are deep neural networks with convolutional structures, which are widely used because of their powerful local feature learning capabilities. Some researchers have also proposed EEGNet compact convolutional neural networks, which can be trained with a limited number of data sets and can produce separable features. However, since EEG signals are essentially time series signals with strong time dependence, CNN cannot fully utilize the time series information in EEG signals. Some researchers have introduced recurrent neural networks (RNN) or long short-term memory networks (LSTM) to process the time correlation of EEG signals. When the amount of data in the data set is very large, the advantages of deep learning are relatively obvious. However, the sample size of EEG signals is limited, which is not enough to train large-scale neural networks, and overfitting problems are prone to occur.

[0004] At present, there are many difficulties in the field of motor imagery EEG signal classification. On the one hand, there is a serious shortage of training samples. Deep learning models are difficult to play their advantages when the amount of data is limited, and overfitting is prone to occur, which affects the generalization ability of the model. On the other hand, EEG signals have the characteristics of high noise and high dimensionality, which makes feature extraction extremely difficult. Traditional methods are difficult to effectively distinguish between useful information and noise in the signal, which further affects the classification accuracy. Therefore, how to effectively extract features and improve signal classification accuracy in the absence of sufficient training samples and high noise and high dimensionality is a key issue that needs to be urgently solved in the current research on motor imagery EEG signal classification. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion, in order to address the problems of lack of sufficient training samples and high noise and high dimension in the field of motor imagery EEG signal classification. The present invention proposes a time series data enhancement technology based on time series decomposition and prediction, which provides more sufficient training samples for the downstream deep learning-based EEG classification model, thereby improving the generalization ability and classification effect of the model. At the same time, in order to address the feature extraction problems caused by high noise and high dimension of motor imagery EEG signals, multimodal feature extraction is performed using time series and time-frequency, further improving the effect of EEG signal classification and recognition.

[0006] In a first aspect, the present invention provides a method for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion, the method comprising the following steps:

[0007] Step S1, obtaining original EEG data of motor imagery;

[0008] Step S2, performing data enhancement on the original EEG data, wherein the data enhancement includes time series data enhancement based on time series decomposition and prediction;

[0009] Step S3: Time series feature extraction:

[0010] S3-1 performs time series segmentation on all channels of the original EEG data and the EEG data after data enhancement to obtain multiple time series blocks; stacks all channel time series blocks in the same time window to obtain new multi-channel time series data;

[0011] S3-2 inputs the multi-channel time series data into the time series feature extractor to extract the time series features of the EEG data;

[0012] Step S4: time-frequency image feature extraction:

[0013] S4-1 performs time-frequency analysis on the original EEG data and the EEG data after data enhancement, and converts the time series data into time-frequency images;

[0014] S4-2 inputs the time-frequency image data into the time-frequency feature extractor to extract the time-frequency image features of the EEG data;

[0015] Step S5, concatenating the time series features and the time-frequency image features to obtain the required multimodal features; finally, inputting the multimodal features into the classifier to obtain the motor imagery classification results.

[0016] Preferably, the raw EEG data in step S1 also needs to be preprocessed, and the preprocessing includes filtering and normalization.

[0017] Preferably, the implementation process of time series data enhancement based on time series decomposition in step S2 includes:

[0018] All channels of the original EEG data are decomposed into time series to obtain multiple decomposition items; then these decomposition items are recombined according to weights to generate a synthetic time series.

[0019] Preferably, the implementation process of time series data enhancement based on time series decomposition in step S2 is specifically:

[0020] The time series of each channel data is decomposed according to the following formula:

[0021] ;

[0022] in represents the kth channel data, express Trend items, express Seasonal items, express A random item; N represents The number of sampling points;

[0023] Recombine the trend item, seasonal item, and random item according to the weight to generate new synthetic EEG data ;

[0024] ;

[0025] , , , ;

[0026] in They represent the weights corresponding to the trend term, seasonal term, and random term respectively.

[0027] Preferably, the implementation process of time series data enhancement based on time series prediction in step S2 includes:

[0028] Construct time series statistical models for all channels of the original EEG data;

[0029] According to the time series statistical model, the fitting value of the original EEG data is obtained, and then the fitting sequence is constructed;

[0030] The original EEG data is fused with the fitting sequence to generate new synthetic EEG data.

[0031] Preferably, the time series statistical model adopts a summed autoregressive moving average model ,in is the order of the autoregressive term, is the number of differences, is the order of the moving average term; specifically:

[0032] Channel data Perform a difference operation, exist After the order difference, it becomes a stationary non-white noise time series , then for the stationary non-white noise time series Establish the corresponding autoregressive moving average model;

[0033] according to The autocorrelation and partial autocorrelation diagrams of Model order determination ;

[0034] according to , and get the fitting sequence ;

[0035] Fusion of original EEG data with fitting sequence to generate new synthetic EEG data ;

[0036] ; , ;

[0037] in Represent the weights corresponding to the original EEG data and the fitting sequence respectively.

[0038] Preferably, the temporal feature extractor in step S3-2 includes an input layer, a network layer, and an output layer:

[0039] The input layer includes a linear mapping layer and a position encoding layer; wherein the linear mapping layer is used for dimension conversion, and the position encoding layer is used for capturing time sequence information between time sequence blocks;

[0040] The network layer adopts the encoder of the Transformer model, which is composed of multiple identical encoder layers stacked together. Each encoder layer consists of two sublayers: a 3-head self-attention layer and a feedforward neural network layer. The outputs of the two sublayers are both residually connected and layer normalized.

[0041] The output layer includes a linear layer.

[0042] Preferably, the time-frequency feature extractor in step S4-2 adopts a convolutional neural network CNN.

[0043] Preferably, in step S4, multiple channel data related to motor imagery are selected from the original EEG data and the EEG data after data enhancement; and time-frequency analysis is performed using continuous wavelet transform whose wavelet mother function is Haar function.

[0044] In a second aspect, the present invention provides a motor imagery EEG signal classification and recognition system, comprising:

[0045] Data acquisition module, responsible for acquiring raw EEG data of motor imagery;

[0046] A data enhancement module is responsible for performing data enhancement on the original EEG data, wherein the data enhancement includes time series data enhancement based on time series decomposition and prediction;

[0047] The time series feature extraction module is responsible for extracting time series features from the original EEG data and the EEG data after data enhancement:

[0048] The time-frequency feature extraction module is responsible for extracting time-frequency image features from the original EEG data and the EEG data after data enhancement;

[0049] The multimodal feature extraction module is responsible for splicing the time series features and the time-frequency image features to obtain the required multimodal features;

[0050] The classification module is responsible for inputting multimodal features into the classifier to obtain the motor imagery classification results.

[0051] The present invention provides a method for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion, which has the following significant beneficial effects:

[0052] 1. The present invention proposes a time series data enhancement technology based on time series decomposition and prediction. In the time series data enhancement of time series decomposition, each channel of the original EEG data is decomposed into time series to obtain trend items, seasonal items and random items, which are then recombined according to specific weights to generate synthetic time series, and the data volume is expanded by adjusting the weights. In the time series data enhancement of time series prediction, a time series statistical model (such as ), the sequence is stabilized by differential operation, the model is determined according to the autocorrelation and partial autocorrelation diagrams, and the fitting sequence is obtained, which is then fused with the original data to generate new synthetic data. Finally, the two enhanced data are mixed to provide more sufficient training samples for the deep learning model, solving the problem of insufficient training samples in the classification of motor imagery EEG signals and improving the generalization ability of the model.

[0053] 2. The present invention uses time series and time-frequency to perform multimodal feature extraction to solve the problem of feature extraction caused by high noise and high dimension of motor imagery EEG signals. In the time series feature extraction, all channels of the original and enhanced EEG data are segmented into time series to obtain multiple time series blocks. The channel time series blocks in the same time window are stacked into new multi-channel time series data, which are input into the time series feature extractor to effectively extract time series features. When extracting time-frequency image features, specific channel data (such as C3, Cz, C4) related to motor imagery are selected, the time series data is converted into time-frequency images, and then the time-frequency image features are extracted by the time-frequency feature extractor. Finally, the time series features and time-frequency image features are spliced ​​to obtain multimodal features, which are input into the classifier to obtain the classification results. Multimodal data fusion and feature enhancement significantly improve the accuracy of the motor imagery classification algorithm.

[0054] 3. In the process of time series feature extraction, the present invention performs time series segmentation on the EEG data and divides the time series into Patch time series blocks. The model can process each block centrally without processing time points one by one, which effectively reduces the dimensions that the model needs to process and the number of parameters, thereby reducing the computational complexity and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the implementation modes of the present invention and the technical solutions in the prior art, the drawings required for use in the description of the implementation modes or the prior art are briefly introduced below.

[0056] Figure 1 It is a flow chart of a method for classifying and identifying motor imagery EEG signals based on data enhancement and multimodal feature fusion provided by an embodiment of the present invention.

[0057] Figure 2 It is the time series decomposition result of the C3 channel EEG data provided by the embodiment of the present invention.

[0058] Figure 3 This is the data enhancement result of the C3 channel based on time series decomposition provided by an embodiment of the present invention.

[0059] Figure 4 It is an autocorrelation and partial autocorrelation diagram of a stationary non-white noise time series provided by an embodiment of the present invention.

[0060] Figure 5This is the data enhancement result based on time series prediction provided by an embodiment of the present invention.

[0061] Figure 6 This is the time series segmentation of multi-channel EEG data provided by an embodiment of the present invention.

[0062] Figure 7 It is a time-frequency diagram of three channels (C3, Cz, C4) provided by an embodiment of the present invention.

[0063] Figure 8 It is a time series feature extractor provided by an embodiment of the present invention.

[0064] Fig. 9 It is a flow chart of time-frequency image feature extraction provided by an embodiment of the present invention.

[0065] Fig.10 This is the experimental paradigm provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0066] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0067] This embodiment provides a method for classifying and recognizing motor imagery EEG signals based on data enhancement and multimodal feature fusion. Figure 1 , the method comprises the following steps:

[0068] Step S1, obtaining the original EEG data of motor imagery; the motor imagery experimental paradigm is shown in Fig.10 ;

[0069] Step S2: perform data enhancement on the original EEG data to expand the training samples required by the deep network model;

[0070] Step S3, extracting time series features;

[0071] Step S4, extracting time-frequency image features;

[0072] Step S5, concatenating the time series features and the time-frequency image features to obtain the required multimodal features; finally, inputting the multimodal features into the classifier to obtain the motor imagery classification results.

[0073] In one preferred implementation, the raw EEG data in step S1 needs to be preprocessed, and the preprocessing includes filtering and normalization. The details are as follows:

[0074] The raw EEG data is filtered using a bandpass filter to remove high-frequency and low-frequency noise interference; in this embodiment, an 8-30 Hz bandpass filter can be selected. The bandpass filter is a Butterworth filter, such as an 8-30 Hz 44-order Chebyshev II bandpass filter, which can filter out high-frequency and low-frequency noise interference.

[0075] The filtered EEG data is then normalized, such as the Max-Min standard method, to map the original EEG data to the [0,1] interval; specifically:

[0076] The multi-channel EEG time series signal can be expressed as the following two-dimensional matrix:

[0077]

[0078] in, It is a two-dimensional matrix that arranges EEG signals in parallel according to the acquisition channels. is the number of channels, is the number of sampling points. , representing 22 EEG channels.

[0079] Perform Max-Min normalization on each channel data, taking the first channel as an example:

[0080]

[0081] The other 21 channel data are normalized to obtain the normalized EEG signal matrix.

[0082]

[0083] Specifically, the data enhancement in step S2 includes time series data enhancement based on time series decomposition and prediction;

[0084] In one preferred implementation, the implementation process of time series data enhancement based on time series decomposition includes:

[0085] All channels of the original EEG data are decomposed into time series to obtain multiple decomposition items; these decomposition items are then recombined according to weights to generate a synthetic time series. Specifically:

[0086] The time series of each channel data is decomposed according to the following formula:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] in represents the kth channel data, express Trend items, express Seasonal items, express A random item; N represents The number of sampling points;

[0093] Then the trend item, seasonal item and random item are recombined according to the weights to generate new synthetic EEG data. ;

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] in They represent the weights corresponding to the trend term, seasonal term, and random term respectively.

[0100] Take the C3 channel EEG data as the first channel data Take this as an example to illustrate: Can be decomposed into trend terms , Seasonal Item and random items ,See Figure 2 :

[0101]

[0102] in:

[0103]

[0104]

[0105]

[0106]

[0107] For other channels of EEG signal data Also make corresponding processing, by adjusting the weight , thereby expanding the 22-channel EEG time series data . The basis of the weight setting strategy is that, according to the research and experimental verification of the characteristics of EEG signals, trend items and seasonal items often contain more effective information related to the motor imagery task, so they are given relatively large weights. By adjusting the weights, the proportion of different components in the synthetic data can be flexibly controlled, thereby achieving the expansion of the data volume. For example, in this embodiment, by carefully setting the weights, the data volume can be expanded to 5 times the original data, and enhanced data for each channel is obtained. In actual operation, it is necessary to determine the optimal weight value through multiple experiments and data analysis based on the specific data characteristics and experimental requirements.

[0108] Figure 3 This is the data enhancement result of the C3 channel based on time series decomposition.

[0109] In one preferred implementation, the implementation process of time series data enhancement based on time series prediction includes:

[0110] Construct time series statistical models for all channels of the original EEG data;

[0111] According to the time series statistical model, the fitting value of the original EEG data is obtained, and then the fitting sequence is constructed;

[0112] The original EEG data is fused with the fitting sequence to generate new synthetic EEG data.

[0113] Preferably, the time series statistical model adopts a summed autoregressive moving average model ( ) model, where is the order of the autoregressive term, which determines the degree of dependence of the model on past data and is determined by analyzing the autocorrelation function (ACF). It is the number of differences, which is used to convert non-stationary time series into stationary series. The difference operation can eliminate the trend and seasonal components in the data and make the data meet the stationary assumption. The determination of the number of differences needs to be combined with the actual data and the unit root test results. It is the order of the moving average term, which reflects the degree of smoothness of the model for random fluctuations and is determined by the partial autocorrelation function (PACF).

[0114] It is a random time series analysis model. Its modeling process is: remove the trend and seasonal components in the time series through differential operation, stabilize the original series, and then establish the corresponding autoregressive moving average model for the stationary non-white noise time series. . The specific modeling process is as follows, mainly including: 1. Model identification and order determination; 2. Model fitting; 3. Model testing; 4. Model optimization.

[0115] Specifically: for channel data Perform a difference operation, exist After the order difference, it becomes a stationary non-white noise time series , then for the stationary non-white noise time series Establish the corresponding autoregressive moving average model.

[0116] according to The autocorrelation and partial autocorrelation plots of Figure 4 ),conduct Model order determination .

[0117] According to the determined Model parameters, using least squares method and other parameter estimation methods to calculate the fitting sequence During the calculation process, the model predicts future data points based on past data points and the determined autoregressive and moving average coefficients, thereby generating a fitted sequence. The fitted sequence reflects the model's ability to learn and capture the trends and patterns of the original data.

[0118] Fusion of original EEG data with fitting sequence to generate new synthetic EEG data , see Figure 5 This fusion strategy aims to make full use of the real information in the original data and the predicted information in the fitting sequence, and introduces more data changes without losing the key features of the original data by setting the weights reasonably, thereby achieving the purpose of data enhancement.

[0119]

[0120]

[0121]

[0122] in Represent the weights corresponding to the original EEG data and the fitting sequence respectively.

[0123] Finally, the enhanced EEG data obtained based on time series decomposition and enhanced EEG data based on time series prediction The enhanced EEG data are mixed to obtain data, and C represents the number of channels. The data enhanced in two different ways are merged according to the corresponding channels to obtain a richer and more diverse data set.

[0124] ;

[0125] Specifically, step S3 includes:

[0126] S3-1 First, all channels of the original EEG data and the EEG data after data enhancement are segmented into time series according to a certain size of window and step size to obtain multiple time series blocks; each time series block captures the information of the local area, allowing the model to focus on the characteristics of different areas. This helps the model better understand the local information of the time series. At the same time, dividing the time series into Patch time series blocks can reduce the dimensions that the model needs to process. The model does not need to process each time point one by one, but focuses on processing each block, which effectively reduces the number of model parameters and reduces the computational complexity.

[0127] Then, all channel time series blocks in the same time window are stacked to obtain new multi-channel time series data.

[0128] See also Figure 6 , with the first channel data For example, according to The sequence length is , by Patch length For time series segmentation, the selection of a Patch length of 256 is the optimization result obtained after a large number of experiments and data analysis. This length can capture local information while ensuring the efficiency and effect of model processing. A smaller Patch length may lead to information loss and fail to fully reflect the local characteristics of EEG signals; while a too large Patch length will increase the computational complexity of the model, reduce the processing speed, and may introduce too much irrelevant information. Two consecutive Patches are processed without overlap and can be divided into 3 Patch time blocks. , , .in

[0129]

[0130]

[0131]

[0132] For other channels of EEG signal data The same time segmentation process is performed. Divide into , , , Divide into , , . For the 22 channel data, the same time series segmentation process can be performed to obtain 66 Patch time series blocks. The two-dimensional matrix , which is converted into The two-dimensional matrix .

[0133]

[0134] S3-2 inputs the multi-channel time series data into the time series feature extractor (TST, Time Series Transformer) to extract the time series features of the EEG data.

[0135] In one preferred embodiment, the temporal feature extractor includes an input layer, a network layer, and an output layer. Figure 8 :

[0136] The input layer includes a linear mapping layer and a position encoding layer; wherein the linear mapping layer is used for dimensional conversion, adjusting the dimension of the input data to a form suitable for subsequent processing. This process involves linear algebra operations such as matrix multiplication. Through a carefully designed mapping matrix, the data dimension is optimized. The position encoding layer uses a learnable position encoding matrix Position Embedding to capture the time sequence information between time series blocks. During the processing, the position encoding vector is added to the input time series block data, so that the model can distinguish time series blocks at different positions, thereby better learning the time series dependencies in the data.

[0137] The network layer adopts the encoder of the Transformer model, which is composed of multiple identical encoder layers stacked together. Each encoder layer contains Multi-Head Attention, Feed Forward, Add&Norm, namely, self-attention layer, feedforward neural network layer, residual connection and layer normalization.

[0138] In the self-attention layer, the Transformer's self-attention mechanism calculates the weights of the EEG features of different time slices to find the dependencies between time steps. Specifically, first, the query vector is obtained by linear transformation at each position of the input sequence. , the key vector , value vector Secondly, and the key vector Then, the softmax function is used to normalize the result. Finally, the normalized score is multiplied by each value vector. , and then calculate the weighted sum, which is the output of the self-attention layer. The outputs of the three subspaces corresponding to the three self-attention heads are merged together. The multi-head self-attention mechanism is calculated by the following formula, where is the scale factor.

[0139]

[0140] The feedforward neural network layer adopts a simple fully connected layer structure, which includes two linear transformations and a ReLU activation function. Its function is to perform feature mapping and dimension transformation on the data and extract more advanced feature representations. The outputs of the two sub-layers are processed by residual connection and layer normalization. The purpose of residual connection is to solve the gradient vanishing problem in deep neural networks, so that the network can better transmit gradient information during training and accelerate convergence. The specific operation is to directly add the input and output of the sub-layer to form a shortcut so that information can be directly transmitted between network layers. Layer normalization is to normalize the input of each sample in each layer so that it has zero mean and unit variance, which helps to improve the stability and generalization ability of the model. Through such a structural design, the encoder of the Transformer model can effectively learn the temporal features in EEG data and provide strong support for subsequent classification and recognition.

[0141] The output layer includes a linear layer, whose main function is to map the high-dimensional features learned by the network layer to a low-dimensional space to obtain the final temporal feature representation. In this embodiment, the output layer maps the feature dimension to a dimension suitable for the classification task, for example, the output dimension is set according to the number of classification categories, so as to connect with the subsequent classifier to realize the classification and recognition of motor imagery EEG signals.

[0142] Specifically, see Fig. 9 , step S4 comprises:

[0143] S4-1 performs time-frequency analysis on the original EEG data and the EEG data after data enhancement, and converts the time series data into time-frequency images;

[0144] This embodiment preferably selects the data of the three channels (C3, Cz, and C4) that are most relevant to motor imagery. This selection is based on in-depth research on the neural mechanism of motor imagery in the brain and analysis of a large amount of experimental data. The C3, Cz, and C4 channels are located in the motor cortex area of ​​the brain. During motor imagery, the neural activities in these areas are closely related to the imagination of limb movement and can more effectively reflect the characteristics of motor imagery tasks. By focusing on the data of these key channels, the amount of data processing can be reduced while improving the accuracy and effectiveness of feature extraction.

[0145] Then, the continuous wavelet transform with the wavelet mother function as the Haar function is used for time-frequency analysis, and the time-frequency images of the three channels are spliced ​​from top to bottom in the order of (C3, Cz, C4), and then the spliced ​​time-frequency images are converted into RGB image, see Figure 7 . The RGB image is used as the input of the subsequent deep network model CNN; specifically:

[0146] CWT analysis is performed on the three channels (C3, Cz, C4) of the initial EEG data and the EEG data after data enhancement to generate a time-frequency image containing time and frequency. The continuous wavelet transform uses the following formula:

[0147]

[0148] in, is the wavelet transform coefficient, is a signal or square integrable function at time t, For telescopic scale, is the translation parameter, for The complex conjugate function of .

[0149] Wavelet mother function is the Haar wavelet function:

[0150]

[0151] S4-2 inputs the time-frequency image data into the time-frequency feature extractor to extract the time-frequency image features of the EEG data.

[0152] In one preferred implementation manner, the time-frequency feature extractor adopts a convolutional neural network (CNN).

[0153] The CNN model includes: convolution layer C1 (convolution kernel size , number of convolution kernels 6), pooling layer S2 (pooling layer size ), convolution layer C3 (convolution kernel size , number of convolution kernels 512), fully connected layer.

[0154] In this embodiment, the details are as follows:

[0155] (1) Input layer: The RGB image of the time-frequency graph obtained by wavelet transform is used as the input of the CNN network (size is );

[0156] (2) Convolutional layer C1, the convolution kernel size in the convolutional layer is , the number of convolution kernels is 6, and the size of the feature map after convolution is ;

[0157] (3) Pooling layer S2, in which the size is selected The kernel is pooled to obtain a feature map size of ;

[0158] (4) Convolution layer C3, the convolution kernel size is , the number of convolution kernels is 512, and the featuremap obtained after convolution is , and get 512 convolution results;

[0159] (5) Linear layer: obtain a 512-dimensional feature vector.

[0160] CNN model training and optimization: In practical applications, CNN models need to be trained to learn the characteristic patterns in time-frequency images. The training process involves selecting appropriate loss functions (such as cross entropy loss functions for multi-classification tasks) and optimization algorithms (such as stochastic gradient descent SGD and its variants Adagrad, Adadelta, Adam, etc.). By inputting a large amount of labeled time-frequency image data into the model, calculating the error between the predicted result and the true label, and then back-propagating the error according to the optimization algorithm, the weight parameters of the model are adjusted so that the model can gradually learn effective feature representations and improve classification accuracy. At the same time, in order to prevent overfitting, regularization techniques (such as L1 and L2 regularization, Dropout, etc.) can also be used to constrain the complexity of the model during training and enhance the generalization ability of the model. In the present invention, through carefully designed experimental procedures and parameter adjustments, the CNN model can effectively extract features related to motor imagery from time-frequency images, providing strong support for the final classification and recognition.

[0161] Specifically, step S5 includes:

[0162] The time series features and time-frequency image features are spliced ​​to obtain the required multimodal features. During the splicing process, it is necessary to ensure the compatibility of the two features in terms of dimension and representation. For example, if the dimension of the time series features is m and the dimension of the time-frequency image features is n, they can be combined into a (m+n)-dimensional multimodal feature vector through a simple vector splicing operation. This splicing method can organically combine the time-dependent information in the time series features and the time-frequency local information in the time-frequency image features, providing the classifier with more comprehensive and rich feature inputs, thereby improving the accuracy of classification and recognition.

[0163] The multimodal features are input into a classifier (such as support vector machine SVM, multi-layer perceptron MLP, etc.) to obtain the motor imagery classification results.

[0164] To verify the effectiveness of the algorithm, the BCI Competition Ⅳ2a dataset provided by Graz University of Technology was used. This dataset consists of 9 subjects providing 4 different motor imagery tasks, including imagining the movements of the left hand, right hand, foot, and tongue. To verify the effectiveness of the algorithm, this research method was modeled and analyzed on the experimental dataset. 80% of the 288 trial data of each subject was used as the training set, and 20% was used as the test set. The cross-validation method is used to calculate the average classification accuracy. The network parts of the time series feature extractor and the time-frequency feature extractor are both trained in batches, with a training batch size of 16 and a training round number of 100. At the same time, the data will be shuffled and verified after each round of training to improve the robustness of the training. The experimental results mainly include four parts: (1) Classification algorithm results based on time series signals and time series feature extractors (TST, Time Series Transformer); (2) Classification algorithm results based on time-frequency images and time-frequency feature extractors (Convolutional Neural Network CNN); (3) Classification algorithm results based on multimodal feature (time series + time-frequency feature) fusion; (4) Classification algorithm based on data enhancement and multimodal feature fusion.

[0165] Table 1 Comparison of experimental accuracy results of each model (%)

[0166]

[0167] For the multi-modal data of motor imagery EEG signals, classification models based on single-modal data and classification models based on multi-modal feature fusion were constructed respectively. As can be seen from Table 1, the average classification accuracy of the two single-modal algorithms (time series data + time series feature extraction, time-frequency image + time-frequency feature extractor) are 72.42% and 71.65% respectively. After feature extraction and fusion of motor imagery EEG multi-modal data, the average classification accuracy of the multi-modal deep learning algorithm is 76.52%. Compared with the single-modal algorithm, it can be improved by 4.1 percentage points. This shows the effectiveness of multi-modal data fusion and feature enhancement in motor imagery classification algorithms.

[0168] The present invention proposes a method for data enhancement of EEG time series data, including time series data enhancement based on time series decomposition and time series data enhancement based on time series prediction, which realizes the data expansion of EEG time series data and effectively expands the training sample size. As can be seen from the table, after the introduction of the data enhancement scheme, the average classification accuracy is 80.07%. Compared with the training data before expansion, it can be increased by 3.55 percentage points. Through the above comparative experiments, it is shown that the data enhancement and multimodal feature fusion method proposed in this study is effective in the modeling process of motor imagery classification algorithm, which can improve the classification effect of the model.

[0169] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for classifying and recognizing motor imagery EEG signals based on data enhancement and multimodal feature fusion, characterized in that: The method comprises the following steps: Step S1, obtaining original EEG data of motor imagery; Step S2, performing data enhancement on the original EEG data, wherein the data enhancement includes time series data enhancement based on time series decomposition and prediction; The implementation process of time series data enhancement based on time series decomposition in step S2 includes: All channels of the original EEG data are decomposed into time series to obtain multiple decomposition items. These decomposition items are then recombined according to weights to generate a synthetic time series. Specifically: The time series of each channel data is decomposed according to the following formula: ； in represents the kth channel data, express Trend items, express Seasonal items, express A random item; N represents The number of sampling points; Recombine the trend item, seasonal item, and random item according to the weight to generate new synthetic EEG data ; ; ; , , ; in Respectively represent the weights corresponding to the trend item, seasonal item, and random item; The implementation process of time series data enhancement based on time series prediction in step S2 includes: Construct time series statistical models for all channels of the original EEG data; According to the time series statistical model, the fitting value of the original EEG data is obtained, and then the fitting sequence is constructed; The original EEG data is fused with the fitting sequence to generate new synthetic EEG data; The time series statistical model adopts the summed autoregressive moving average model ,in is the order of the autoregressive term, is the number of differences, is the order of the moving average term; specifically: Channel data Perform a difference operation, exist After the order difference, it becomes a stationary non-white noise time series , then for the stationary non-white noise time series Establish the corresponding autoregressive moving average model ; according to The autocorrelation and partial autocorrelation diagrams of Model order determination ; according to , and get the fitting sequence ; Fusion of original EEG data with fitting sequence to generate new synthetic EEG data ; ; ; ; in They represent the weights corresponding to the original EEG data and the fitting sequence respectively; Step S3: Time series feature extraction: S3-1 performs time series segmentation on all channels of the original EEG data and the EEG data after data enhancement to obtain multiple time series blocks; stacks all channel time series blocks in the same time window to obtain new multi-channel time series data; S3-2 inputs the multi-channel time series data into the time series feature extractor to extract the time series features of the EEG data; Step S4: time-frequency image feature extraction: S4-1 performs time-frequency analysis on the original EEG data and the EEG data after data enhancement, and converts the time series data into time-frequency images; S4-2 inputs the time-frequency image data into the time-frequency feature extractor to extract the time-frequency image features of the EEG data; Step S5, concatenating the time series features and the time-frequency image features to obtain the required multimodal features; finally, inputting the multimodal features into the classifier to obtain the motor imagery classification results.

2. The method according to claim 1, characterized in that: In step S1, the original EEG data also needs to be preprocessed, and the preprocessing includes filtering and normalization.

3. The method according to claim 1, characterized in that: The temporal feature extractor in step S3-2 includes an input layer, a network layer, and an output layer: The input layer includes a linear mapping layer and a position encoding layer; wherein the linear mapping layer is used for dimension conversion, and the position encoding layer is used for capturing time sequence information between time sequence blocks; The network layer adopts the encoder of the Transformer model, which is composed of multiple identical encoder layers stacked together. Each encoder layer consists of two sublayers: a 3-head self-attention layer and a feedforward neural network layer. The outputs of the two sublayers are both residually connected and layer normalized. The output layer includes a linear layer.

4. The method according to claim 1, characterized in that: The time-frequency feature extractor in step S4-2 uses a convolutional neural network CNN.

5. The method according to claim 1, characterized in that: In step S4, multiple channel data related to motor imagery are selected from the original EEG data and the EEG data after data enhancement; and time-frequency analysis is performed using continuous wavelet transform whose wavelet mother function is the Haar function.

6. A motor imagery EEG signal classification and recognition system implementing the method described in any one of claims 1 to 5, characterized in that include: Data acquisition module, responsible for acquiring raw EEG data of motor imagery; A data enhancement module is responsible for performing data enhancement on the original EEG data, wherein the data enhancement includes time series data enhancement based on time series decomposition and prediction; The time series feature extraction module is responsible for extracting time series features from the original EEG data and the EEG data after data enhancement: The time-frequency feature extraction module is responsible for extracting time-frequency image features from the original EEG data and the EEG data after data enhancement; The multimodal feature extraction module is responsible for splicing the time series features and the time-frequency image features to obtain the required multimodal features; The classification module is responsible for inputting multimodal features into the classifier to obtain the motor imagery classification results.

Images