A dynamic spatio-temporal feature enhancement network model for motor imagery classification

Abstract

The application discloses a dynamic space-time feature enhancement network model for motor imagery classification, and relates to the technical field of motor imagery, comprising a dynamic space-time feature enhancement module and a space-time convolution module, wherein the dynamic space-time feature enhancement module comprises a multi-scale time convolution layer, which is used for extracting alpha and beta band features in an MI related electroencephalogram (EEG) signal and reserving the original EEG signal as a baseline feature layer; the dynamic space-time feature enhancement module further comprises a grouping space convolution layer, which is used for extracting multi-level space features and preventing overfitting through weight constraint; and the space-time convolution module is used for further extracting space-time features and performing classification. The technical scheme of the application combines a DSTA and a space-time convolution (STC) module, and in 10-fold cross-validation, compared with ShallowConvNet, the average accuracy of the DSTA-Net is increased by 6.29% (p<0.01), 3.05% (p<0.01), 5.26% (p<0.01) and 2.25% on BCI-IV-2a, OpenBMI, CASIA and stroke patient data sets respectively.

Description

Technical Field

[0001] This invention relates to the field of motion imagery technology, and in particular to a dynamic spatiotemporal feature enhancement network model for motion imagery classification. Background Technology

[0002] Brain-computer interfaces (BCIs) acquire signals from the central nervous system via sensors when a user performs mental tasks or receives stimuli. These signals are then encoded into instructions for interaction with a computer, thereby enabling the restoration, enhancement, or replacement of neural functions. Among various physiological signal acquisition technologies, non-invasive electroencephalography (EEG) has been widely adopted in BCI applications due to its low cost, rapid response, and ease of deployment, such as neurofeedback training, wheelchair control, drone navigation, and virtual reality. Compared to steady-state visual evoked potentials (SSVEPs) and event-related potentials (ERPs), motor imagery (MI) has attracted significant attention due to its potential to enhance neural plasticity and promote rehabilitation.

[0003] MI-EEG signals are characterized by non-stationarity, low signal-to-noise ratio, and high intra-class variability, making spatiotemporal feature extraction difficult. Traditional machine learning relies on handcrafted features (such as band power and cospatial patterns), but this requires specialized knowledge and has poor generalization ability. Deep learning overcomes this bottleneck through automatic feature learning: for example, DeepConvNet uses deep convolutions to extract complex spatiotemporal features, ShallowConvNet focuses on shallow band power features, and EEGNet combines spatial-frequency convolutions, performing efficiently in small sample scenarios.

[0004] Single-scale convolutional networks struggle to fully extract the spatiotemporal information of EEG. Multi-scale methods enhance feature representation by using convolutional kernels of different sizes, but too many kernels can lead to increased computational complexity, information redundancy, and optimization difficulties, while too few kernels may limit feature representation, resulting in the loss of key information and a decrease in generalization ability. Summary of the Invention

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is to provide a dynamic spatiotemporal feature enhancement network model for motor image classification, including a dynamic spatiotemporal feature enhancement module and a spatiotemporal convolution module, wherein the dynamic spatiotemporal feature enhancement module includes a multi-scale temporal convolution layer for extracting α and β frequency band features from MI-related electroencephalogram (EEG) signals and retaining the original EEG signal as a baseline feature layer;

[0006] The dynamic spatiotemporal feature enhancement module also includes a grouped spatial convolutional layer for extracting multi-level spatial features and preventing overfitting through weight constraints.

[0007] The spatiotemporal convolution module is used to further extract spatiotemporal features and perform classification.

[0008] Furthermore, the multi-scale temporal convolutional layer comprises multiple levels of one-dimensional temporal convolutional kernels, wherein the temporal length of the i-th convolutional kernel is defined as: Used to dynamically capture spatiotemporal features of different frequency ranges in electroencephalogram (EEG) signals;

[0009] pass and Determine the kernel size to match the feature ranges of the α and β bands in the motion imagery (MI) task;

[0010] An original signal compensation layer is introduced into the multi-scale temporal convolutional layer to reduce the distortion of the feature matrix caused by multi-scale convolution;

[0011] The outputs of the original signal compensation layer, α-band feature layer, and β-band feature layer The data are then spliced ​​together to generate an enhanced spatiotemporal feature representation.

[0012] Furthermore, the constrained grouped spatial convolutional layer includes three groups of spatial convolutional modules, each of which uses a grouped spatial convolutional operation with a kernel size of (Nc,1), where Nc represents the number of EEG signal channels;

[0013] Each spatial convolution module generates 10 convolution kernels. The three sets of output feature maps are concatenated along the convolution kernel dimension to obtain a feature matrix Xspatial with shape Bs×Nf×1×T, where Bs represents the batch size of 16, Nf=30 represents the total number of convolution kernels, and T represents the time dimension.

[0014] The maximum norm constraint is applied to the weight vector of each convolution kernel, and the L2 norm is re-normalized to make the L2 norm value of the weight vector less than 2.

[0015] For each feature channel of the feature matrix Xspatial, BatchNorm2d normalization and Swish activation function transformation are performed sequentially.

[0016] The activated feature matrix is ​​reshaped into a three-dimensional tensor structure, maintaining its channel dimension consistent with the spatial dimension of the original EEG signal, thus forming a feature matrix that integrates spatiotemporal features.

[0017] Furthermore, the spatiotemporal convolution module includes

[0018] Temporal convolutional layer: 60 filters are used to perform temporal convolution on the input EEG signal, with a kernel size of k = (1, 25) to extract temporal features;

[0019] Spatial convolutional layer: The output of the temporal convolutional layer is convolved in the spatial dimension with a kernel size of k = (60,1), which expands the number of feature map channels to 120;

[0020] Batch normalization layer: Performs BatchNorm2d normalization on the output of the spatial convolutional layer to reduce internal covariate bias;

[0021] Square Enhancement Layer: The normalized features are squared element-wise using a custom SquareLayer module to enhance feature separability.

[0022] Average pooling layer: AvgPool2d is used to downsample the features in the time dimension, with a pooling kernel size of p = (1, 100) and a stride of s = (1, 10).

[0023] Logarithmic Transformation Layer: The LogLayer module performs a logarithmic transformation on the pooled features to optimize the feature representation;

[0024] Random dropout layer: Apply the Dropout operation to the features with probability p=0.5 to randomly deactivate some neurons to prevent overfitting;

[0025] Classification convolutional layer: The 120 input channels are mapped to the output class through temporal convolution, the kernel size is (1, 88), and the LogSoftmax activation function is used to output the classification probability;

[0026] Loss calculation: Optimize model parameters based on negative log-likelihood loss (NLLLoss).

[0027] Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0028] The Dynamic Spatiotemporal Feature Enhancement Network (DSTA-Net) proposed in this application combines DSTA and Spatiotemporal Convolution (STC) modules. In the DSTA module, multi-scale temporal convolution kernels are designed for the α and β frequency bands of MI neurophysiological features, while the original EEG is used as a baseline feature layer to preserve original information. Grouped spatial convolutions extract multi-level spatial features, combined with weight constraints to prevent overfitting. The spatial convolution kernel maps EEG channel information to a new spatial domain, achieving further feature extraction through dimensionality transformation. The STC module performs feature extraction and classification.

[0029] DSTA-Net was evaluated on three public datasets and applied to a self-collected stroke patient dataset. In 10-fold cross-validation, DSTA-Net achieved average accuracy improvements of 6.29% (p < 0.01), 3.05% (p < 0.01), 5.26% (p < 0.01), and 2.25% compared to ShallowConvNet on the BCI-IV-2a, OpenBMI, CASIA, and stroke patient datasets, respectively. In hold-out validation, DSTA-Net achieved average accuracy improvements of 3.99% (p < 0.01) and 4.2% (p < 0.01) compared to ShallowConvNet on the OpenBMI and CASIA datasets, respectively. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the dynamic spatiotemporal feature enhancement network model for motion image classification as described in this invention.

[0032] Figure 2 This is the experimental paradigm and EEG preprocessing diagram of the stroke dataset in this invention;

[0033] Figure 3 This is a graph showing the individual classification accuracy of the OpenBMI dataset (HO) of this invention.

[0034] Figure 4 This is a confusion matrix diagram of different algorithms for the OpenBMI and CASIA datasets under HO analysis in this invention;

[0035] Figure 5 This is a DeepLift visualization of different algorithms for Subject 1 in the BCI-IV-2a dataset of this invention (with the left hand as the target label and the right hand as the reference label);

[0036] Figure 6 The t-SNE visualization of subject 1 in the BCI-IV 2a dataset of this invention under different algorithms (labels 0, 1, 2 and 3 represent the left hand, right hand, foot and tongue in imagined movement, respectively);

[0037] Figure 7The figure shows the ablation experiment results of four datasets under the cross-validation (CV) mode of this invention (based on paired t-test, statistical significance is marked with ** (p<0.01));

[0038] Figure 8 The DeepLift visualization of the stroke dataset (subject 1, accuracy 96.33%, and subject 3, accuracy 42.67%) using DSTA-Net (with left and right hands as target labels and resting task as reference label) is shown below:

[0039] Figure 9 CSP topographic maps of the left and right hand MI tasks of patients 1 and 3 in the α and β frequency bands of this invention. Detailed Implementation

[0040] This invention proposes a dynamic spatiotemporal feature enhancement network model for motion image classification, aiming to present a dynamic spatiotemporal feature enhancement network model that combines DSTA and spatiotemporal convolution (STC) modules.

[0041] The specific structure of the dynamic spatiotemporal feature enhancement network model for motion image classification proposed in this invention will be described below in a specific embodiment:

[0042] Example 1:

[0043] A dynamic spatiotemporal feature enhancement network model for motion image classification includes a dynamic spatiotemporal feature enhancement module and a spatiotemporal convolution module, wherein:

[0044] The dynamic spatiotemporal feature enhancement module includes a multi-scale temporal convolutional layer, which is used to extract α and β frequency band features from MI-related electroencephalogram (EEG) signals and retain the original EEG signal as a baseline feature layer.

[0045] The dynamic spatiotemporal feature enhancement module also includes a grouped spatial convolutional layer for extracting multi-level spatial features and preventing overfitting through weight constraints.

[0046] The spatiotemporal convolution module is used to further extract spatiotemporal features and perform classification.

[0047] Furthermore, the multi-scale temporal convolutional layer comprises multiple levels of one-dimensional temporal convolutional kernels, wherein the temporal length of the i-th convolutional kernel is defined as: Used to dynamically capture spatiotemporal features of different frequency ranges in electroencephalogram (EEG) signals;

[0048] pass and Determine the kernel size to match the feature ranges of the α and β bands in the motion imagery (MI) task;

[0049] An original signal compensation layer is introduced into the multi-scale temporal convolutional layer to reduce the distortion of the feature matrix caused by multi-scale convolution;

[0050] The outputs of the original signal compensation layer, α-band feature layer, and β-band feature layer The data are then spliced ​​together to generate an enhanced spatiotemporal feature representation.

[0051] Furthermore, the constrained grouped spatial convolutional layer includes three groups of spatial convolutional modules, each of which uses a grouped spatial convolutional operation with a kernel size of (Nc,1), where Nc represents the number of EEG signal channels;

[0052] Each spatial convolution module generates 10 convolution kernels. The three sets of output feature maps are concatenated along the convolution kernel dimension to obtain a feature matrix Xspatial with shape Bs×Nf×1×T, where Bs represents the batch size of 16, Nf=30 represents the total number of convolution kernels, and T represents the time dimension.

[0053] The maximum norm constraint is applied to the weight vector of each convolution kernel, and the L2 norm is re-normalized to make the L2 norm value of the weight vector less than 2.

[0054] For each feature channel of the feature matrix Xspatial, BatchNorm2d normalization and Swish activation function transformation are performed sequentially.

[0055] The activated feature matrix is ​​reshaped into a three-dimensional tensor structure, maintaining its channel dimension consistent with the spatial dimension of the original EEG signal, thus forming a feature matrix that integrates spatiotemporal features.

[0056] Furthermore, the spatiotemporal convolution module includes

[0057] Temporal convolutional layer: 60 filters are used to perform temporal convolution on the input EEG signal, with a kernel size of k = (1, 25) to extract temporal features;

[0058] Spatial convolutional layer: The output of the temporal convolutional layer is convolved in the spatial dimension with a kernel size of k = (60,1), which expands the number of feature map channels to 120;

[0059] Batch normalization layer: Performs BatchNorm2d normalization on the output of the spatial convolutional layer to reduce internal covariate bias;

[0060] Square Enhancement Layer: The normalized features are squared element-wise using a custom SquareLayer module to enhance feature separability.

[0061] Average pooling layer: AvgPool2d is used to downsample the features in the time dimension, with a pooling kernel size of p = (1, 100) and a stride of s = (1, 10).

[0062] Logarithmic Transformation Layer: The LogLayer module performs a logarithmic transformation on the pooled features to optimize the feature representation;

[0063] Random dropout layer: Apply the Dropout operation to the features with probability p=0.5 to randomly deactivate some neurons to prevent overfitting;

[0064] Classification convolutional layer: The 120 input channels are mapped to the output class through temporal convolution, the kernel size is (1, 88), and the LogSoftmax activation function is used to output the classification probability;

[0065] Loss calculation: Optimize model parameters based on negative log-likelihood loss (NLLLoss).

[0066] Example 2:

[0067] A dynamic spatiotemporal feature enhancement network model for motion image classification includes a dynamic spatiotemporal feature enhancement module and a spatiotemporal convolution (STC) module, wherein:

[0068] The dynamic spatiotemporal feature enhancement module includes a multi-scale temporal convolutional layer, which is used to extract α and β frequency band features from MI-related electroencephalogram (EEG) signals and retain the original EEG signal as a baseline feature layer.

[0069] The dynamic spatiotemporal feature enhancement module also includes a grouped spatial convolutional layer for extracting multi-level spatial features and preventing overfitting through weight constraints.

[0070] The spatiotemporal convolution module is used to further extract spatiotemporal features and perform classification.

[0071] 1. The architecture of DSTA-Net is shown in Table 1:

[0072] Table 1 shows the architecture of DSTA-Net:

[0073]

[0074]

[0075] 1.1 Dynamic Spatiotemporal Feature Enhancement Module:

[0076] The model's input determines the upper limit of feature extraction and classification performance. Therefore, enhancing the representation of raw EEG data helps improve model performance. MI contains rich spatiotemporal feature information. To address this, the DSTA module is proposed to enhance the spatiotemporal feature representation of MI. In the DSTA module, dynamic multi-scale temporal convolutional layers aim to improve the EEG signal representation while preserving as much original information as possible. Given the high temporal resolution of EEG signals, the first layer of DSTA-Net includes dynamic multi-scale temporal convolutional layers along the time dimension, composed of multi-scale one-dimensional temporal convolutional kernels. The length of each convolutional kernel is set to a specific proportion of the EEG sampling frequency fs, defined as follows: γ i ∈R, where i represents the layer in the multi-scale temporal convolution. If the dynamic temporal layer contains N layers, then the value of i ranges from 1 to N. Therefore, the temporal kernel size of the i-th layer... Defined as:

[0077] From a frequency perspective, different temporal kernel sizes can enrich the model's learning representation of dynamic frequencies in EEG. For example, in the sentiment decoding task, Tsception uses a dynamic temporal kernel of length [1 / 2, 1 / 4, 1 / 8] × fs, which can capture dynamic frequency features above [2, 4, 8] Hz. Research has found that MI activity is mainly concentrated in the α and β frequency ranges. To capture features in these frequency bands, the convolutional kernel size was determined through formula derivation. The sampling frequency was set to:

[0078] Substituting Formula 1 into Formula 2, we get:

[0079] Solve the inequalities for the α and β frequency bands respectively:

[0080] The results show:

[0081] After comparing the candidate proportions [1 / 2, 1 / 4, 1 / 8, 1 / 16], it was found that... It matches well with the α band. Matching the β band, [1 / 8, 1 / 16] was chosen as the small-scale time kernel. Furthermore, to reduce the distortion of the feature matrix relative to the original EEG signal after multi-scale convolution, an original signal layer was added on top of the multi-scale convolution to compensate for the information loss introduced during the convolution process.

[0082] From a temporal perspective, multi-scale time kernels can capture both short-term and long-term temporal patterns. Let X = [X0, X1, ..., X...]. n ], where X n ∈R c×tX represents the EEG signal, n represents the number of trials, c represents the number of EEG channels, and t represents the sampling point for time information. The output of each layer of the multi-scale temporal convolution is defined as:

[0083]

[0084] p i =int(γ i ×fs / 2)

[0085] Where p i The padding is indicated by concatenating the outputs along the kernel dimension after convolution operations in a dynamic multi-scale temporal convolutional neural network, as shown in the following formula:

[0086]

[0087] in and These represent the original signal layer, the α-band feature layer, and the β-band feature layer, respectively.

[0088] The constrained grouped spatial convolutional layer combines spatial convolution, batch normalization, and the Swish activation function. The grouped spatial convolution uses a kernel size of (Nc, 1), where Nc represents the number of EEG channels. Considering... and Three sets of spatial convolutions were designed, each representing a different feature layer. Each set generates 10 convolutional kernels, which are then concatenated along the kernel dimension. The resulting feature matrix Xspatial has a shape of Bs×Nf×1×T, where Bs represents the batch size of 16, Nf represents the total number of convolutional kernels generated by the three sets of convolutions, and T represents the time dimension. To avoid overfitting, the weights are constrained using the maximum norm. The weight vector is renormalized so that its L2 norm is less than 2. Then, BatchNorm2d is used to normalize each feature channel, combined with the Swish activation function, to improve the model's non-linear expressive power.

[0089] After passing through the constrained grouped spatial convolutional layer, the output tensor is... Where Nf represents the number of filters (convolution kernels) learned during spatial convolution. Each filter can be viewed as a mapping, transforming EEG spatial information into a new feature space, effectively capturing rich spatial features from the data. To further enhance spatiotemporal feature extraction capabilities, the output is reshaped into... This reshaping allows the model to preserve the differences in spatial features between channels while effectively learning temporal patterns. X reshape The data structure is consistent with the original EEG, so it is regarded as a feature matrix that integrates spatial and temporal information.

[0090] 1.2 Spatiotemporal Convolution Module:

[0091] The spatiotemporal convolution module adopts a network structure similar to ShallowConvNet, but with fine-tuned filter numbers and kernel sizes. It consists of temporal convolution, spatial convolution, BatchNorm2d, Avgpool2d, and logarithmic transform layers.

[0092] Temporal convolution uses 60 filters and the kernel size is k = (1, 25).

[0093] Spatial convolutions use a kernel of size k = (60, 1) to expand the feature map to 120 channels. The output of the spatial convolution is normalized using BatchNorm2d to reduce internal covariate bias and improve training stability. A custom SquareLayer module squares the elements to enhance feature separability. Temporal downsampling is performed using AvgPool2d with a kernel size p = (1, 100) and a stride s = (1, 10). LogLayer applies a logarithmic transformation to the features to enhance representation learning. Dropout is applied with a probability of p = 0.5 to randomly deactivate a subset of neurons. Finally, a temporal convolution maps the 120 input channels to the output class with a kernel size of (1, 88). A LogSoftmax activation function is used, followed by a negative log-likelihood loss (NLLLoss) as the loss function.

[0094] 1.3 Evaluation Indicators:

[0095] The main evaluation metrics of this application include classification accuracy, kappa coefficient, and confusion matrix, used to evaluate the performance of DSTA-Net. Interpretability metrics include CSP, t-SNE, and DeepLIFT, used to analyze the model's feature representation and decision-making process. Specifically, this application employs a rescaling-based DeepLift method to interpret the feature contribution distribution of DSTA-Net. DeepLIFT calculates attribution scores through backpropagation, quantitatively evaluating the impact of each input feature on the neural network's output decision. For example, when analyzing the right-handed MI task, the right-handed MI task is used as the target input, while the left-handed MI task or rest task is used as the baseline input, implemented using the DeepLift module in the Captum toolkit.

[0096] 2. Experiment:

[0097] Establish experimental dataset

[0098] The self-collected stroke patient dataset has been approved by the Ethics Committee of Shanghai Second Rehabilitation Hospital (Approval No.: 2023-10-01). Before participating in the experiment, each participant signed an informed consent form and completed a detailed questionnaire regarding age, sex, dominant hand, previous BCI experience, and health status. Data were collected from 8 chronic right-handed stroke patients (5 males, 3 females; mean age = 52 years; Brunnstrom stage III-V; right arm paralysis). The experimental paradigm and EEG preprocessing are as follows: Figure 2 As shown, the system consists of four modules, each containing 75 trials. Tasks include left-hand MI, right-hand MI, and rest tasks. Task types are randomly assigned within each module, with 25 trials for each task type. Each trial includes a 3-second cue time, a 4-second task time, and a 2-second rest time. Considering that some participants may be unfamiliar with the BCI system, they are encouraged to attempt the tasks multiple times to reach the motor threshold without actual physical movement.

[0099] EEG data were recorded using a Neuracle 64-channel wireless amplifier at a sampling rate of 1000 Hz, and the MI task was written using Eprime 3.0. Data preprocessing was performed using the MATLAB plugin EEGLAB, including downsampling to 250 Hz, 0.5–40 Hz FIR bandpass filtering, rereference, AAR-based EEG artifact removal, and noise removal combining ICA and MARA. Finally, the data were randomly shuffled and split into two label-equalized sessions for analysis. Twenty electrodes in the motion region (including FC-5 / 3 / 1 / z / 2 / 4 / 6, C-5 / 3 / 1 / z / 2 / 4 / 6, and CP-5 / 3 / 1 / 2 / 4 / 6) were selected for analysis. Table 2 provides detailed information on the various datasets.

[0100] Table 2. Dataset Information:

[0101]

[0102] The BCIIV-2a dataset used 22 Ag / AgCl electrodes to collect EEG data from 9 participants at a sampling rate of 250 Hz. Each participant performed two sessions on different days, each session consisting of four types of MI tasks: left hand, right hand, tongue, and both feet. Each task had 72 trials, with each trial containing 4 seconds of EEG data.

[0103] The OpenBMI dataset used 62 electrodes to collect EEG data from 54 healthy participants. The raw sampling rate was 1000 Hz, downsampled to 250 Hz. Each participant underwent two MI-EEG recording sessions, each session including left-hand and right-hand MI tasks. Each task had 100 trials, each containing 4 seconds of task-related EEG data. Twenty electrodes in the motor region (including FC-5 / 3 / 1 / 2 / 4 / 6, C-5 / 3 / 1 / z / 2 / 4 / 6, and CP-5 / 3 / 1 / z / 2 / 4 / 6) were selected for analysis.

[0104] The CASIA dataset collected EEG data from 25 healthy participants with no prior experience with MI-BCI. Using 64 electrodes at a sampling rate of 1000 Hz, participants performed three MI tasks: “rest,” “hand,” and “elbow,” with 300 trials per task. Each trial lasted 4 seconds. Data preprocessing used the EEGLAB toolbox, including common average reference (CAR), 0.1–40 Hz bandpass filtering, baseline drift removal, AAR-based artifact removal, and downsampling to 200 Hz. The dataset focused on classifying the “hand” and “elbow” tasks using data from 20 motor region electrodes, consistent with the OpenBMI dataset. The dataset contains 15 sessions, divided into two balanced sessions, each containing 150 trials for both left-hand and right-hand tasks.

[0105] DSTA-Net was evaluated using 10-fold cross-validation (CV) and hold-out (HO) analysis. While some studies use only Session 1 data to eliminate inter-session variability, given the potential for overfitting due to limited data in deep learning models, we chose to use data from both Session 1 and Session 2 for cross-validation. In 10-fold cross-validation, 9 folds were used for training and 1 fold for testing to ensure class balance and improve the model's generalization ability. In the hold-out setting, Session 1 was used for training and Session 2 for testing to evaluate the model's ability to extract generalizable features for cross-session classification.

[0106] 3. Training strategies:

[0107] All algorithms employ a consistent training process to ensure model robustness and generalization. The model uses the Adam optimizer with default parameters (learning rate = 0.001, β1 = 0.9, β2 = 0.999) and uses log-cross-entropy loss to guide gradient updates. A two-stage training strategy is employed. In Stage 1, the training data is divided into training and validation sets. If the validation accuracy does not improve for 200 consecutive epochs, an early stopping strategy is applied, and the parameters are restored to the state where the validation accuracy was highest. In Stage 2, the optimal model from Stage 1 is used, combined with data from both the training and validation sets, and training continues until the validation loss falls below the training loss from Stage 1. To ensure convergence, Stage 1 and Stage 2 training are limited to 1500 epochs and 600 epochs, respectively. For cross-validation (CV), within a 9-fold training framework, 1 fold of the training data is selected as the validation set; for hold-out (HO) analysis, 20% of the training data is reserved as the validation set. In both settings, test data is not used in the training process. This comprehensive strategy effectively prevents overfitting and improves model performance and generalization ability. All models are implemented using the PyTorch framework.

[0108] 4. Results:

[0109] 4.1 Comparison with baseline:

[0110] Table 3 summarizes the average classification results compared to baseline methods on four MI datasets (BCI-IV-2a, OpenBMI, CASIA, and the stroke patient dataset). Furthermore, Table 4 shows the average classification accuracy and kappa coefficients for different algorithms and datasets in the HO analysis. In both the CV and HO analyses, DSTA-Net outperformed the baseline algorithms in terms of classification accuracy.

[0111] Table 3. Average classification accuracy (CV) on different algorithms and datasets:

[0112]

[0113] Bold numbers indicate the highest accuracy. Paired t-test significance is marked with * (p<0.05) and ** (p<0.01).

[0114] Table 4. Average classification accuracy and Kappa coefficient on different algorithms and datasets (HO):

[0115]

[0116] This application conducted computer vision (CV) tests on four MI datasets. On the BCI-IV-2a dataset, DSTA-Net achieved an accuracy of 85.70% across four tasks, significantly outperforming baseline algorithms (p<0.05), especially showing a clear advantage compared to ShallowConvNet, DeepConvNet, and EEGITNet (p<0.01). On the OpenBMI dataset, DSTA-Net achieved an average accuracy of 77.18%, significantly surpassing ShallowConvNet and EEGITNet (p<0.01), and also showing a significant improvement over DeepConvNet (p<0.05). Although the difference with EEGNet was not statistically significant, DSTA-Net still achieved a 2.75% improvement (77.18% vs. 74.43%). On the CASIA dataset, DSTA-Net achieved the highest average accuracy of 67.51%, significantly outperforming ShallowConvNet (p<0.05) and DeepConvNet (p<0.01). On the stroke patient dataset, the DSTA-Net algorithm achieved a top classification accuracy of 67.50% across the three-class classification task, outperforming EEGNet and EEGITNet (p<0.01). This suggests that DSTA-Net may be more effective in capturing relevant features from the stroke patient dataset, potentially implying superior feature extraction capabilities and stronger model robustness.

[0117] Considering the impact of sample size on statistical power and the reliability of results, HO tests were performed on larger OpenBMI and CASIA datasets. Figure 3As shown, the HO analysis results of different algorithms on the OpenBMI dataset are presented. DSTA-Net achieves an average accuracy of 65.57%, surpassing all baseline algorithms, showing a significant improvement compared to ShallowConvNet (p<0.05), and an even more significant improvement compared to DeepConvNet (p<0.01). Furthermore, DSTA-Net's kappa coefficient reaches 0.3115, the highest among all algorithms, reflecting stronger classification consistency compared to ShallowConvNet (kappa = 0.2317), DeepConvNet (kappa = 0.2070), EEGNet (kappa = 0.2804), and EEGITNet (kappa = 0.2756). On the CASIA dataset, DSTA-Net achieves an average accuracy of 62.28%, consistently outperforming baseline methods. Its kappa coefficient is 0.2456, which also outperforms other algorithms, including ShallowConvNet (kappa = 0.1616), DeepConvNet (kappa = 0.1949), EEGNet (kappa = 0.1977), and EEGITNet (kappa = 0.2189). Detailed classification results for the four MI datasets can be found in the supplementary materials. Figure 4 As shown (horizontal axis represents the predicted result, vertical axis represents the actual label: PL / PR (predicted left / right), AL / AR (actual left / right), PE / PH (predicted elbow / hand), AE / AH (actual elbow / hand)), the confusion matrix of HO analysis on the OpenBMI and CASIA datasets is presented. To evaluate the classification performance on these datasets, this application analyzed the confusion matrix for all subjects. On these datasets, DSTA-Net exhibits superior overall performance, achieving the highest correct classification rate and the lowest false classification rate.

[0118] On the OpenBMI dataset, DSTA-Net achieved correct classification rates of 64.28% and 66.87% for the two classes, respectively. On the CASIA dataset, these rates were 63.39% and 61.17%, respectively. Compared to other models, DSTA-Net exhibited significantly fewer misclassifications. EEGNet demonstrated some competitiveness on both the OpenBMI and CASIA datasets, but its robustness was slightly weaker than DSTA-Net, with a relatively higher misclassification rate. Models such as DeepConvNet and ShallowConvNet had lower classification accuracy and higher misclassification rates, which was more pronounced on the CASIA dataset, reflecting their limitations in feature extraction and classification. Overall, these results highlight DSTA-Net's superior capabilities in EEG feature extraction and classification, making it the most robust model on both datasets.

[0119] To further evaluate DSTA-Net, this application analyzed the recognition accuracy of the OpenBMI dataset under the CV strategy, comparing the top 25% and bottom 25% of test takers. As shown in Table 5, DSTA-Net achieved an accuracy of 95.21% in the top 25% of test takers and 58.38% in the bottom 25% of test takers, demonstrating strong classification ability, especially with a significant advantage among high-performing test takers.

[0120] Table 5. Performance comparison of the top 25% and bottom 25% of subjects on the OpenBMI dataset:

[0121]

[0122] 5. Ablation experiment:

[0123] This application's DSTA-Net consists of DSTA and STC modules. The DSTA module specifically includes multi-scale temporal convolutions and grouped spatial convolutions. To evaluate the contributions of these components, ablation experiments using the OpenBMI dataset and HO analysis were conducted, and the results are shown in Table 6. Scheme A represents the complete DSTA-Net, Scheme B replaces the dynamic multi-scale layer with the original signal as input to DSTA-Net, and Scheme C uses the original signal as input, performing feature extraction and classification only through the STC module. Under HO analysis, Scheme A achieved an average classification accuracy of 65.57%, Scheme B 62.19%, and Scheme C 61.58%. Scheme A has a 3.38% higher accuracy than Scheme B (p<0.01), highlighting the crucial role of the dynamic multi-scale layer. Scheme B has a 0.61% higher accuracy than Scheme C, indicating that constrained spatial grouped convolutions with dimensionality transformation enhance the model's ability to extract multi-scale temporal features. Furthermore, Scheme A improved performance by 3.99% compared to Scheme C (p<0.01), demonstrating that the DSTA module helps enhance feature extraction from the raw EEG signal, thereby improving classification performance.

[0124] Table 6. Subject-level classification accuracy of the OpenBMI dataset (HO):

[0125]

[0126] MTC stands for Multi-Scale Temporal CNN, GSC stands for Grouped Spatial CNN, and STC stands for SpatioTemporal CNN. A checkmark (√) indicates that the module is included, while a cross (×) indicates that the module is excluded.

[0127] 6. Explainability and visualization:

[0128] To evaluate the discriminative power of features extracted by DSTA-Net and baseline methods, this study employs the t-distributed random neighborhood embedding (t-SNE) algorithm to visualize the high-dimensional latent features of the classification layer in a two-dimensional space. Using the multi-class BCI-IV-2a dataset as an example, this application conducts experiments using data from Subject 1 to illustrate the effectiveness of t-SNE in evaluating feature separability. Furthermore, this application applies higher-order (HO) analysis to evaluate the classification performance of multiple models. The classification accuracies for Subject 1 are as follows: DSTA-Net 81.25%, ShallowConvNet 79.86%, DeepConvNet 77.43%, EEGNet 75.35%, and EEGITNet 75.34%.

[0129] Figure 5 The DeepLift results of different algorithms on Subject 1 of the BCI-IV-2a dataset are visualized.

[0130] To facilitate comparisons across feature dimensions, the attribution scores were normalized. The attribution distribution of the DSTA-Net model is mainly concentrated near the right C4 region and exhibits significant asymmetry, consistent with the neurophysiological characteristic of left-hand MI activating the right brain. In contrast, while the attribution distribution of the contrast algorithm contributes to the area near C4, it is more dispersed in other brain regions, which may reduce the classification performance of these models.

[0131] Figure 6 The visualization results of t-SNE for several techniques are presented, demonstrating the discriminative potential of the extracted features. DSTA-Net and ShallowConvNet show the best inter-class separation performance, with significantly reduced inter-class overlap, exhibiting excellent discriminative ability. DeepConvNet and EEGNet perform slightly worse, with some overlap in point cloud distribution, but still maintain reasonable inter-class separability. In contrast, EEGITNet shows the most significant inter-class overlap, resulting in relatively weaker feature discriminative ability and clustering performance.

[0132] 7. Discussion:

[0133] The effect of DSTA-Net:

[0134] MI-BCI shows promise in promoting neural plasticity and offers great hope for stroke rehabilitation. However, non-stationarity and high intraclass variability pose significant challenges to MI decoding. Traditional single-scale CNN methods have achieved some success in MI decoding, but their performance still needs further improvement. The introduction of multi-scale CNNs has improved decoding performance, but the choice of the number and size of convolutional kernels affects the decoding results. Considering the neurophysiological and spatiotemporal characteristics of the α and β bands of MI, this application proposes DSTA-Net from the perspective of enhancing spatiotemporal feature representation. To verify the performance of DSTA-Net, this application tested it on three public MI datasets and a self-collected stroke patient dataset.

[0135] Regarding decoding performance, Table 3 shows that in the CV analysis, DSTA-Net improved the average accuracy by 6.29% (p<0.01), 3.05% (p<0.01), 5.26% (p<0.01), and 2.25% compared to ShallowConvNet on the BCI-IV-2a, OpenBMI, CASIA, and stroke patient datasets, respectively. Table 4 shows that in the HO analysis, DSTA-Net improved the average accuracy by 3.99% (p<0.01) and 4.2% (p<0.01) compared to ShallowConvNet on the OpenBMI and CASIA datasets, respectively. (Based on Table 4 and...) Figure 4 DSTA-Net outperforms other algorithms in both kappa coefficient and accuracy, indicating that it has better classification consistency and can provide more stable and reliable results. Figure 5 The performance differences of DSTA-Net on data of varying quality were further evaluated. For the top 25% of subjects, DSTA-Net performed exceptionally well, achieving an average recognition accuracy of 95.21%. However, for the bottom 25% of subjects, DSTA-Net's accuracy was 58.38%, slightly lower than EEGNet by 1.64%. This is because extracting motion intent features from the data of the bottom 25% of subjects is more difficult, and increased noise interference makes it difficult for the DSTA module to achieve significant improvements.

[0136] In terms of ablation experiments, Table 6 highlights the significant contributions of key modules in DSTA-Net: the dynamic multi-scale temporal convolutional neural network, constrained spatial grouping convolutions with dimensionality transformation, and spatiotemporal convolutions. On the OpenBMI dataset, DSTA-Net achieves the highest average accuracy (65.57%), outperforming schemes B (62.19%) and C (61.58%), and improving upon scheme C by 3.99% (p<0.01). This underscores the synergistic effect of its modules in enhancing EEG feature representations. Figure 5 and Figure 6The results show that DSTA-Net exhibits the most significant inter-class separability and the least overlap, reflecting its superior performance in feature extraction and classification.

[0137] To further investigate the statistical differences in the CV analysis, this application performed a paired t-test on the results, such as... Figure 7 As shown, on all four MI datasets, DSTA-Net consistently outperformed scheme C in average recognition rate (P<0.01). Although the improvement on the patient dataset did not reach statistical significance, the recognition performance was still improved by 2.25%.

[0138] Interpretability and visualization of stroke patient data:

[0139] In clinical applications, analyzing brain activity patterns in stroke patients helps to gain a deeper understanding of the neural mechanisms involved in the MI task. This understanding is crucial for improving the accuracy and reliability of BCI systems in clinical settings. In this study, we applied DeepLift to evaluate the contribution of each EEG channel to the model's decision-making, comparing the target task with a reference task (e.g., resting state). Furthermore, we used CSP to investigate patients' activation patterns. The analysis included subjects with the highest classification accuracy (96.33%) (subject 1) and the lowest (42.67%) in the stroke patient dataset.

[0140] Figure 8 This study presents DeepLift results using the DSTA-Net algorithm for stroke patients 1 and 3 (both males with right-sided hemiplegia; patient 1's Upper Extremity Functional Assessment Scale (UFMA) score was 61, and patient 3's was 33). In this study, left-hand and right-hand tasks were designated as target tasks, and the resting task was used as the reference task. Results show that patient 1 demonstrated significant discriminative ability across tasks, achieving a classification accuracy of 96.33%, while patient 3's accuracy was only 42.67%. Figure 8 As shown, during the left-hand MI task, Patient 1 exhibited significant and concentrated activation in the right motor cortex, while the left hemisphere showed partial compensatory activation. Conversely, during the right-hand MI task, the left brain region showed clear and concentrated activation, while the right brain region showed partial compensatory activation. In contrast, Patient 3 showed lower discriminability between the left-hand and right-hand MI tasks, with activation in both hemispheres. DeepLift results indicate that during the left-hand task, Patient 3's right hemisphere was more active.

[0141] Since the activation levels and regions identified by DeepLift largely depend on the model's feature extraction capabilities, this application further analyzes the activation levels and regions during the MI task from the perspective of the raw EEG signal. This analysis employs the classic CSP feature. Considering the high variance of CSP and its susceptibility to noise artifacts, this application performs meticulous preprocessing on the raw signal, such as... Figure 2 As shown, CSP analysis was performed on 100 left-hand tasks and 100 right-hand tasks for each patient to maximize the variance difference between the two types of tasks.

[0142] Figure 9 The topographic activation patterns in the α (8–13 Hz) and β (13–30 Hz) bands are shown for patients 1 and 3 during left-hand and right-hand MI tasks. For patient 1, during the left-hand MI task, the left hemisphere showed significant α-band suppression (indicated in blue), while the right hemisphere showed significant β-band activation (indicated in red). Conversely, during the right-hand MI task, the right hemisphere showed no significant α-band suppression, while the left hemisphere showed clear β-band activation. These results indicate that patient 1 exhibited significant event-related desynchronization / event-related synchronization (ERD / ERS) in both the α and β bands. In contrast, patient 3 showed lower discriminability between the left-hand and right-hand MI tasks. Specifically, during the left-hand MI task, the left hemisphere showed no significant α-band suppression, while the right hemisphere showed both β-band activation (red) and α-band suppression (blue). Similarly, during the right-hand MI task, the right hemisphere showed no significant α-band suppression, while the left hemisphere showed clear β-band activation. Therefore, patient 3 exhibited ERD / ERS in the β band, while the effect in the α band was not significant. Finally, based on Figure 5 , Figure 8 and Figure 9 Based on the brain region activation locations and regions shown in the data, this application initially found that DeepLIFT has more stable feature interpretability than the common spatial pattern (CSP) and shows more concentrated brain region activation in high-precision data.

[0143] Conclusion: In this study, we proposed DSTA-Net for MI classification. The DSTC module performs dynamic multi-scale feature extraction and representation in the temporal and spatial domains, achieving satisfactory recognition results on multiple datasets while demonstrating strong interpretability.

[0144] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic spatiotemporal feature enhancement network model for motion image classification, characterized in that, It includes a dynamic spatiotemporal feature enhancement module and a spatiotemporal convolution module. The dynamic spatiotemporal feature enhancement module includes a multi-scale temporal convolutional layer, which is used to extract α and β frequency band features from MI-related EEG signals and retain the original EEG signal as baseline features. The dynamic spatiotemporal feature enhancement module also includes a grouped spatial convolutional layer for extracting enhanced spatiotemporal features. Multi-level spatial features are used to form a feature matrix that integrates spatiotemporal features; and overfitting is prevented through a weight constraint mechanism. The spatiotemporal convolution module is used to further extract the spatiotemporal features of the feature matrix integrating spatiotemporal features and perform classification; The multi-scale temporal convolutional layer comprises multiple levels of one-dimensional temporal convolutional kernels, wherein the first... The time length of a convolutional kernel is defined as follows: It is used to dynamically capture the spatiotemporal characteristics of different frequency ranges in electroencephalogram (EEG) signals; pass and Determine the kernel size to match the feature ranges of the α and β bands in the motion visualization task; An original signal compensation layer is introduced into the multi-scale temporal convolutional layer to reduce the distortion of the feature matrix caused by multi-scale convolution; The outputs of the original signal compensation layer, α-band feature layer, and β-band feature layer The data are then spliced ​​together to generate an enhanced spatiotemporal feature representation.

2. The dynamic spatiotemporal feature enhancement network model for motion image classification according to claim 1, characterized in that, The constrained grouped spatial convolutional layer consists of three groups of spatial convolutional modules. Each group of spatial convolutional modules uses a grouped spatial convolutional operation with a kernel size of (Nc,1), where Nc represents the number of EEG signal channels. Each spatial convolution module generates 10 convolution kernels. The three sets of output feature maps are concatenated along the convolution kernel dimension to obtain a feature matrix Xspatial with shape Bs×Nf×1×T, where Bs represents the batch size of 16, Nf=30 represents the total number of convolution kernels, and T represents the time dimension. The maximum norm constraint is applied to the weight vector of each convolution kernel, and the L2 norm is re-normalized to make the L2 norm value of the weight vector less than 2. For each feature channel of the feature matrix Xspatial, BatchNorm2d normalization and Swish activation function transformation are performed sequentially. The activated feature matrix is ​​reshaped into a three-dimensional tensor structure, maintaining its channel dimension consistent with the spatial dimension of the original EEG signal, thus forming a feature matrix that integrates spatiotemporal features.

3. The dynamic spatiotemporal feature enhancement network model for motion image classification according to claim 1, characterized in that, The spatiotemporal convolution module includes Temporal convolutional layer: 60 filters are used to perform temporal convolution on the input EEG signal, with a kernel size of k=(1,25), to extract temporal features; Spatial convolutional layer: Performs spatial dimension convolution on the output of the temporal convolutional layer, with a kernel size of [size missing]. =(60,1), which expands the number of feature map channels to 120; Batch normalization layer: Performs BatchNorm2d normalization on the output of the spatial convolutional layer to reduce internal covariate bias; Square Enhancement Layer: The normalized features are squared element-wise using a custom SquareLayer module to enhance feature separability. Average pooling layer: AvgPool2d is used to downsample the features in the time dimension, and the pooling kernel size is [size missing]. =(1,100), step size is =(1,10); Logarithmic Transformation Layer: The LogLayer module performs a logarithmic transformation on the pooled features to optimize the feature representation; Random drop-out layer: based on probability =0.5 Apply Dropout operation to the features to randomly disable some neurons to prevent overfitting; Classification convolutional layer: The 120 input channels are mapped to the output class through temporal convolution, the kernel size is (1, 88), and the LogSoftmax activation function is used to output the classification probability; Loss calculation: Optimize model parameters based on negative log-likelihood loss.

Images