Rehabilitation lower limb exoskeleton electroencephalogram decoding method based on ensemble learning

Abstract

The invention discloses a rehabilitation lower limb exoskeleton electroencephalogram decoding method based on integrated learning, and relates to the technical field of electroencephalogram decoding, and the method comprises the steps: obtaining an EEG signal of a motor cortex of a user through an electroencephalogram device; performing shallow layer processing on the EEG signal to obtain a first type of features; constructing a deep processing network model of the EEG signals, and performing deep processing on the EEG signals to obtain a second type of features; combining the first type of features and the second type of features, training a classifier, and decoding the motion intention of the EEG signal; and performing visual feedback on the user according to the decoded motion intention, and then executing the decoded motion on the lower limb exoskeleton equipment to obtain proprioceptive feedback. The superficial neural network and the deep neural network are combined to improve the performance of lower limb movement intention, continuous motor imagery experiments are carried out on lower limb exoskeleton equipment, continuous EEG electroencephalogram signals are collected, and an experiment normal form of online vision and proprioceptive feedback is designed.

Description

[0001] This application is a divisional application. The application number of the original application is 202311366447.6, the application date is October 20, 2023, and the name of the invention is "An integrated rehabilitation lower limb exoskeleton EEG decoding method and system." Technical Field

[0002] The present invention relates to the technical field of electroencephalogram decoding, and more particularly to an electroencephalogram decoding method for a lower limb rehabilitation exoskeleton based on ensemble learning. Background Art

[0003] Traditional rehabilitation training, such as physical therapy and occupational therapy, has been effective in restoring function, but the process is time-consuming and relies on the professional skills of physical therapists. With the development of robotics technology, lower limb exoskeleton robots have become an emerging treatment strategy that replaces traditional labor-intensive therapies. These exoskeleton devices can be used to train, enhance, assist or replace human movement, thereby enhancing walking ability.

[0004] During the rehabilitation process, the power robot can provide high-intensity rehabilitation training and objective and quantifiable indicators. However, whether the patient actively participates in rehabilitation training will affect the smooth progress of the rehabilitation process and is closely related to the final rehabilitation effect. In order to improve the rehabilitation effect, a series of improvements have been made. For example, devices equipped with virtual reality technology such as Lokomat provide patients with a more fascinating and motivating training experience to enhance the patient's sense of immersion, thereby improving the training effect. In addition, other rehabilitation interfaces such as forearm crutches, wrist-worn devices, joysticks and buttons are also provided, but these devices are only suitable for patients with normal upper limb function.

[0005] As a natural way of interaction, brain-computer interface (BMI) provides patients with a new way to communicate and control external devices. However, the performance of brain-computer interface (BMI) is affected by many factors, such as the noise of EEG signals, high sensitivity to intra-trial data differences, and the need to rely heavily on manual feature selection. The introduction of a deep learning algorithm (DeepconvNet) improves its performance by batch normalization, exponential linear units, and clipping training strategies, and its performance is better than that of FBCSP (Filter Bank Common Spatial Pattern). The advantage of FBCSP is that it can extract features using frequency band information, while ConvNets do not have a dedicated feature extraction process and the features used are obtained during training. Convolutional networks also perform better than EEGNet in other BMI paradigms, such as sensorimotor rhythm (SMR), P300 visual evoked potential, error-related negativity (ERN), and movement-related cortical potential (MRCP).

[0006] Other advanced deep learning algorithms, such as Transformer and Temporal Convolutional Network (TCN), have also been applied to motor imagery EEG classification and visualization. However, research on online decoding using these advanced deep learning algorithms has not yet been involved.

[0007] Therefore, how to provide a rehabilitation lower limb exoskeleton EEG decoding method based on ensemble learning for online and offline decoding of lower limb exoskeletons, enhance the sense of control over lower limb movements by providing visual and proprioceptive feedback, and thus improve the ability to generate reliable and accurate brain signals is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0008] In view of this, the present invention provides a rehabilitation lower limb exoskeleton EEG decoding method based on ensemble learning, which combines shallow and deep neural networks to improve the performance of lower limb movement intention, conducts continuous motor imagery experiments on lower limb exoskeleton devices, collects continuous EEG signals, and designs an experimental paradigm of online visual and proprioceptive feedback.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: an integrated lower limb rehabilitation exoskeleton EEG decoding method, comprising:

[0010] Obtain EEG signals from the user's motor cortex through an electroencephalogram device;

[0011] Performing shallow processing on the EEG signal to obtain a first type of feature;

[0012] Constructing a deep processing network model of the EEG signal, performing deep processing on the EEG signal to obtain the second type of features;

[0013] The first type of features and the second type of features are combined, and a classifier is trained to decode the movement intention of the EEG signal;

[0014] The user is given visual feedback based on the decoded movement intention, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0015] The first type of features are superficial features, and the second type of features are deep features. Through proprioceptive feedback, users can sense the position and movement of their body parts and get a more interactive experience. Users can adjust their movements in a timely manner based on their own performance, thereby increasing their participation and interest.

[0016] Preferably, shallow processing of the EEG signal comprises:

[0017] Filtering and wavelet transforming the EEG signal to obtain different frequency components of the EEG signal;

[0018] The different frequency components are processed by cospatial pattern analysis to separate the EEG signal into additional subcomponents with the largest variance differences between the different categories;

[0019] The features of different frequency components that have been processed by common space pattern analysis are combined to obtain the first type of features; the first type of features are shallow features.

[0020] Preferably, the EEG signal of the user's motor cortex includes: offline sequence data, first online sequence data and second online sequence data;

[0021] Training a deep processing network model of the EEG signal using the offline sequence data;

[0022] The deep processing network model of the EEG signal is tested using the first online sequence data and the second online sequence data.

[0023] Preferably, the deep processing network model of the EEG signal includes: a deep convolution layer, a first temporal convolution layer, a second temporal convolution layer, a self-attention layer and a TCN layer;

[0024] Reducing the number of trainable parameters by using deep convolution to learn spatial filters through the deep convolution layer;

[0025] Extracting feature maps of EEG signals in different frequency bands through the first temporal convolutional layer;

[0026] Extracting feature maps representing data of different time scales through the second temporal convolutional layer;

[0027] Determine the most effective motor imagery information through the self-attention layer;

[0028] All available temporal information is captured through the TCN layer, and the extracted deep features are output after flattening and classified.

[0029] Preferably, the self-attention layer adopts a multi-head attention mechanism, performs a single attention head in different spaces, maps Q, K and V to different spaces through linear transformation, concatenates the attention results of different spaces, performs a linear transformation and maps them to the output space to obtain the final result.

[0030] Preferably, the process of concatenating the attention results of different spaces and performing a linear transformation to map them to the output space is as follows:

[0031] Apply the projection matrix of each attention head to the inputs Q, K and V respectively to obtain h different queries Q, keys K and values ​​V; the parameter matrix of the projection matrix includes

[0032] Perform self-attention calculation on each attention head to produce the output of h attention heads, the formula is as follows:

[0033]

[0034] Where, i∈(1,h); and Represents the projection matrix of each attention head applied to the inputs Q, K, and V respectively;

[0035] The outputs of these attention heads are concatenated together and weighted to perform a linear transformation to obtain the final output. The formula is as follows:

[0036] MultiHead(Q,K,V)=Concat(head1,…,head h )W O ;

[0037] Among them, W O represents the weight matrix.

[0038] Preferably, all available temporal information is captured through the TCN layer, and the extracted deep features are output after flattening and classified, including:

[0039] Preprocess the EEG signals; for example, filtering and denoising, to reduce the impact of noise on classification performance.

[0040] Cutting the EEG signal into time windows and marking a corresponding movement intention category for each time window;

[0041] Construct a TCN model, wherein the TCN model consists of multiple temporal convolutional layers, each of which uses convolution kernels of different sizes to capture features of different time scales;

[0042] Apply global average pooling or global maximum pooling to capture the information of the entire sequence;

[0043] Connect the pooling output to the fully connected layer to perform the classification task and output predictions for different motion intent categories;

[0044] The TCN model is trained using the labeled training data, and the TCN model is optimized using a loss function and an optimization algorithm.

[0045] Preferably, the acquisition of offline sequence data includes:

[0046] Preset the first prompt time, rest time period, visual prompt time period, motor imagery time period and second prompt time;

[0047] The user is prompted through a display screen, and the user starts from the first prompt time, goes through a rest period, and then indicates the movement direction through a visual prompt. The user performs offline training of motor imagery according to the visual prompt, and a second prompt is performed after the motor imagery period ends; the acquisition of offline sequence data of one trial ends;

[0048] The first online sequence data is based on visual feedback, and an online test of visual feedback is performed during the motor imagery time period to obtain discrete visual feedback;

[0049] The second online sequence data is based on proprioceptive feedback, and visual feedback is performed during the movement imagery time period, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0050] Preferably, the method further comprises: using LOSO cross validation evaluation technology to perform performance evaluation;

[0051] Using accuracy and Cohen's Kappa coefficient as performance indicators, as shown in formula (1) and formula (2), the performance of the algorithm is evaluated by statistically analyzing the Cohen's Kappa coefficient k;

[0052]

[0053] Among them, TP represents a true positive example of a certain class, FP represents a false positive example, FN represents a false negative example, and TN represents a true negative example;

[0054]

[0055] Among them, P o Represents the classification accuracy, P e Represents the results of random classification.

[0056] Preferably, an integrated lower limb rehabilitation exoskeleton EEG decoding system comprises:

[0057] EEG equipment, used to obtain EEG signals from the user's motor cortex;

[0058] A first processing module, used for performing shallow processing on the EEG signal to obtain a first type of feature;

[0059] A second processing module is used to construct a deep processing network model of the EEG signal, perform deep processing on the EEG signal, and obtain a second type of feature;

[0060] A decoding module is used to concatenate the first type of features and the second type of features, and send them to a trained classifier (the first sequence of EEG signals is used to train the classifier) ​​to decode the movement intention of the EEG signal;

[0061] The feedback module is used to provide visual feedback to the user based on the decoded movement intention, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0062] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses an EEG decoding method for a rehabilitation lower limb exoskeleton based on ensemble learning, including: obtaining an EEG signal of a user's motor cortex through an EEG device; shallowly processing the EEG signal to obtain a first type of feature; constructing a deep processing network model of the EEG signal, deeply processing the EEG signal to obtain a second type of feature; merging the first type of feature and the second type of feature, and training a classifier to decode the movement intention of the EEG signal; providing visual feedback to the user based on the decoded movement intention, and then executing the decoded movement on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0063] The present invention combines shallow and deep neural networks to improve the performance of lower limb motor intention. Continuous motor imagery experiments were conducted on lower limb exoskeleton devices independently developed by the laboratory, and continuous EEG brain wave signals were collected, and an experimental paradigm of online visual and proprioceptive feedback was designed. Three thinking tasks (left limb, right limb and rest) and two thinking tasks (motor intention and rest) were classified and performance evaluated. Unlike traditional machine learning or convolutional neural network technology, the present invention connects the spatial-spectral characteristics of wavelets with the temporal convolutional network feature representation that incorporates a multi-head attention mechanism, thereby obtaining performance significantly higher than the chance level, indicating the feasibility of applying integrated methods to brain-controlled lower limb rehabilitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0065] Figure 1 The present invention provides an experimental flow chart of the embodiment of the present invention.

[0066] FIG. 2( a ) is a flowchart of an online test based on visual feedback provided in an embodiment of the present invention.

[0067] FIG2( b ) is a flowchart of an online test based on proprioceptive feedback provided in an embodiment of the present invention.

[0068] Figure 3 A flow chart of shallow processing of EEG signals provided in an embodiment of the present invention.

[0069] Figure 4 A flow chart of deep processing of EEG signals provided in an embodiment of the present invention.

[0070] Figure 5 A schematic diagram of the overall structure of the Transformer provided in an embodiment of the present invention.

[0071] Figure 6 Schematic diagram of the attention mechanism provided for an embodiment of the present invention.

[0072] Figure 7 A schematic diagram of multi-head attention provided for an embodiment of the present invention.

[0073] Figure 8 A schematic diagram of the CN residual network structure provided in an embodiment of the present invention.

[0074] Fig. 9 A schematic diagram of a dilated convolution structure provided in an embodiment of the present invention.

[0075] Fig.10 A schematic diagram of the EEGNet structure provided by an embodiment of the present invention.

[0076] Fig.11 Schematic diagram of the average confusion matrix of the ATCNet and reproduced models provided in the embodiments of the present invention.

[0077] FIG. 12( a ) is a schematic diagram showing how the average EEG signal of the Cz channel changes over time under visual feedback provided by an embodiment of the present invention.

[0078] 12( b ) is a schematic diagram of the change over time of the average EEG signal of the Cz channel under proprioceptive feedback provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The embodiment of the present invention discloses an integrated rehabilitation lower limb exoskeleton EEG decoding method and system. It includes: acquiring EEG signals; shallowly processing the EEG signals to obtain first-class features; deeply processing the EEG signals to obtain second-class features; merging the first-class features and the second-class features to decode the movement intention of the EEG signal; providing visual feedback to the user based on the decoded movement intention, and then executing the decoded movement on the lower limb exoskeleton device to obtain proprioceptive feedback. The embodiment of the present invention combines shallow and deep neural networks to improve the performance of lower limb movement intention, conducts continuous movement imagery experiments on the lower limb exoskeleton device, collects continuous EEG brain wave signals, and designs an experimental paradigm for online visual and proprioceptive feedback.

[0081] In a specific embodiment of the present invention, the embodiment of the present invention discloses an integrated rehabilitation lower limb exoskeleton EEG decoding method, including: an integrated rehabilitation lower limb exoskeleton EEG decoding method, including: obtaining the EEG signal of the user's motor cortex through an EEG device; shallowly processing the EEG signal to obtain a first type of feature; constructing a deep processing network model of the EEG signal, deeply processing the EEG signal to obtain a second type of feature; the first type of feature is the shallow feature, and the second type of feature is the deep feature; the first type of feature and the second type of feature are merged, and a classifier is trained to decode the movement intention of the EEG signal; visual feedback is given to the user according to the decoded movement intention, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0082] Specifically, 8 subjects (4 males and 4 females, with an average age of 27.13 years old) were selected in the present embodiment. They had all conducted brain-computer interface (BCI) related experiments and had performed left-hand and right-hand motor imagery tasks, but had no experience in using similar equipment for lower limb proprioceptive motor imagery. All had normal or corrected-to-normal vision and did not report any known history of neurological or psychiatric diseases.

[0083] In terms of mechanical structure, the embodiment of the present invention uses a laboratory-made bionic three-degree-of-freedom lower limb exoskeleton device, which uses a novel parallel mechanism for gait rehabilitation. In terms of algorithm, it combines the feedback control framework with the repeated learning algorithm to enhance the anti-interference ability of the neural controller. The lower limb exoskeleton device is integrated with the EEG recording system, which can perform the rehabilitation process in an active and natural way, such as Figure 1As shown. It includes EEG recording and robotic systems. The display and lower limb exoskeleton device provide visual and proprioceptive feedback respectively; during the experiment, the subject stood on the exoskeleton, with his legs tied to the bionic knee joints, wearing a wire suit, and his back hanging on the weight-reducing device. Visual cues were presented through a display (50 inches) at a distance of about 2 meters, and control instructions were sent using Python scripts.

[0084] For ease of understanding, Figure 1 The process information shown is explained as follows, among which, CNN (Convolutional Neural Networks); MSA (multi-head self-attention); TCN (Temporal Convolutional Network); WPD (Wavelet Packet Decomposition); CSP (Common Spatial Pattern); Concat; HO (post-hoc tests); LOSO (Leave-One-Out Cross-Validation).

[0085] The EEG equipment selected was the NeuSenW system of the domestic Boricon company for wireless EEG signal recording. The system has 64 built-in scalp electrodes, arranged according to the 10 / 20 international system, with a sampling rate of 1000Hz. In order to obtain EEG signals from the motor cortex, the peripheral channels were removed and the following 26 EEG electrodes were retained (FT7, FT8, FCz, FC1, FC2, FC3, FC4, FC5, FC6, T7, T8, Cz, C1, C2, C3, C4, C5, C6, TP7, TP8, CP1, CP2, CP3, CP4, CP5, CP6). The unipolar EEG signal was amplified and transmitted to a computer (operating system: Windows 10, CPU: Intel Core i7, memory: 16 GB) through the intelligent synchronization center. Another multi-parameter synchronizer was used to generate various trigger signals to synchronize the EEG acquisition system. The synchronization of EEG signals and lower limb exoskeleton devices is sent to the upper computer interface of the lower limb exoskeleton device with the help of TCP / IP protocol, and the upper computer sends gait information to the lower computer through several USB serial ports. The experimental protocol is designed based on the trigger interface provided by the manufacturer and the EEGLAB open source signal processing environment.

[0086] The lower limb exoskeleton device is calibrated before each experiment, including the maximum speed and acceleration of the actuator, to achieve a comfortable training experience. The subjects cannot move any part of their body during the recording, especially they cannot grab the handrails or shake the rope. For safety reasons, if the interaction force is greater than the allowable value, you can press the emergency stop button next to it for an emergency stop, and a dedicated person will monitor the experimental process at all times. In addition, the preparation of the EEG acquisition system includes the positioning of the active electrodes and the application of the conductive gel, and the impedance of the electrodes must be checked to ensure that it is below 10k. The time required to disassemble the experimental setup is about 20 minutes.

[0087] Specifically, the EEG signal of the user's motor cortex includes: offline sequence data, first online sequence data and second online sequence data;

[0088] Training a deep processing network model of the EEG signal using the offline sequence data;

[0089] The deep processing network model of the EEG signal is tested using the first online sequence data and the second online sequence data.

[0090] Specifically, the acquisition of the offline sequence data includes:

[0091] Preset the first prompt time, rest time period, visual prompt time period, motor imagery time period and second prompt time;

[0092] The user is prompted through a display screen, and the user starts from the first prompt time, goes through a rest period, and then indicates the movement direction through a visual prompt. The user performs offline training of motor imagery according to the visual prompt, and a second prompt is performed after the motor imagery period ends; the acquisition of offline sequence data of one trial ends;

[0093] The first online sequence data is based on visual feedback, and an online test of visual feedback is performed during the motor imagery time period to obtain discrete visual feedback;

[0094] The second online sequence data is based on proprioceptive feedback, and visual feedback is performed during the movement imagery time period, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0095] Specifically, as shown in Figure 2(a) and Figure 2(b), three training paradigms are shown. Sequence 1 is offline recording, sequence V is visual feedback, and sequence P is proprioceptive feedback. Each participant performed three recording sequences according to the timeline shown in Figure 2(a) and Figure 2(b), namely offline sequence data, first online sequence data, and second online sequence data. Each sequence was separated by two days. The experiment was divided into three sequences, each of which included 5 rounds, and each round contained 60 trials. These 60 trials were divided into 30 left feet and 30 right feet, and were performed in a random order to prevent the adaptation effect. The offline sequence data was used for offline training, and the first online sequence data and the second online sequence data were used for online testing. The offline sequence was first performed. Each trial started with a fixed cross displayed on the screen as the first prompt time, which lasted for 4 seconds as the resting period. Subsequently, a cursor-like visual prompt was displayed on the screen as a visual prompt period, which lasted for 2 seconds to indicate the direction of movement. After that, a blank screen was displayed on the screen for 4 seconds as a motor imagery period to perform the motor imagery task. Each trial lasted for a total of 10 seconds. At the end of each trial, a blue line was displayed as a second cue time, indicating the end of the trial, with a random interval of 1.5-3.5 seconds to prevent fatigue. Offline sequence data recording takes about 50 minutes to complete.

[0096] Due to the limited number of subjects, the embodiment of the present invention does not perform cross-validation because the results may not be statistically reliable. Therefore, the first online sequence data and the second online sequence data are designed as online tests, wherein the first online sequence data uses visual feedback and the second online sequence data uses proprioceptive feedback. In both feedback modes, the subject will also have visual feedback similar to the offline sequence, and the movement of the progress bar still starts from the middle position, but the direction and speed at this time are determined by the output of the classifier at each time point (such as posterior probability, classification results, etc.), and it does not necessarily reach both ends according to the user's performance. There are two situations at the end of each trial. The first is that the brain-computer interface ends after decoding the subject's intention, and the second is a timeout, that is, the intention is not decoded and ends after a 5-second timeout.

[0097] However, in the second sequence, the subjects will receive additional discrete visual feedback as the progress bar flashes several times, and in the third sequence, the subjects will receive proprioceptive feedback after returning to the middle starting position after performing the decoded movement (movement imagination of the left or right leg) on ​​the lower limb exoskeleton device. The total time required for the second and third sequences is approximately 60 and 70 minutes, respectively. Through proprioceptive feedback, the subjects can perceive the position and movement of their body parts and get a more interactive experience. The subjects can adjust their movements in time according to their own performance, thereby increasing the subjects' participation and interest.

[0098] Specifically, shallow processing of the EEG signal includes:

[0099] Filtering and wavelet transforming the EEG signal to obtain different frequency components of the EEG signal;

[0100] The different frequency components are processed by cospatial pattern analysis to separate the EEG signal into additional subcomponents with the largest variance differences between the different categories;

[0101] The features of different frequency components that have been processed by common space pattern analysis are combined to obtain the first type of features; the first type of features are shallow features.

[0102] Specifically, Python MNE (Electroencephalogram (EEG) and Magnetoencephalogram (MEG) Analysis and Visualization Tool) 1.3, Tensorflow 2.11, and Matlab R2015b (by MathWorks, USA) combined with EEGLAB 13.5.4b toolbox were used for the online and offline data processing of EEG signals. Figure 1 A flow chart of a signal processing method is shown, including a deep processing part and a shallow processing part.

[0103] Specifically, Figure 3 As shown in the figure, the specific architecture of the shallow part is shown. Including notch filtering, spatial filtering, wavelet transform and co-spatial pattern, the EEG data is filtered using an 8th order Butterworth bandpass filter with cutoff frequencies of 0.5Hz and 50Hz, and a 4th order 50Hz notch filter is used. Wavelet transform is used to decompose the signal into a set of wavelets. Compared with the traditional Fourier transform that captures global frequency information, wavelet decomposition can play a role in EEG applications because the signal has characteristic oscillations with short intervals. The signal is decomposed at 5 levels using the 'db4' wavelet, and 8 frequency band coefficients are selected from the range of 4-32Hz. Subsequently, the co-spatial pattern (CSP) is used to separate the signal into additional sub-components that have the largest variance difference between different categories. The number of CSP components is 4.

[0104] The embodiment of the present invention combines wavelet transform and common spatial pattern analysis (Common Spatial Patterns, CSP) to extract and enhance relevant features in the signal, especially in the field of brain-computer interface (Brain-Computer Interface, BCI) and biomedical signal processing. This combination can help obtain better feature representation, thereby improving the performance of classification, recognition or parsing tasks. First, preprocessing is performed. First, the original signal is subjected to wavelet transform to obtain different frequency components of the signal. This can help capture the characteristics of the signal in different frequency ranges. For data with more important spatial information such as EEG signals, CSP preprocessing can be performed to enhance task-related signal components. CSP can generate a set of spatial filters for maximizing the variance difference of different categories. Then feature extraction is performed. For the signal after wavelet transform, CSP is applied in each frequency subband to enhance task-related features. For each frequency subband, CSP can generate a set of spatial filters to project the signal into a specific space so that the variance of some categories is maximized and the variance of other categories is minimized. Feature combination is then performed to combine the features processed by common spatial pattern analysis in different frequency subbands to obtain a comprehensive feature representation. This can be achieved by simply concatenating feature vectors, averaging, etc. Finally, classification is performed and the combined features are used to classify and identify the motion intention category.

[0105] Specifically, the deep processing network model of the EEG signal includes: a deep convolution layer, a first temporal convolution layer, a second temporal convolution layer, a self-attention layer and a TCN layer; the deep convolution layer is used to learn the spatial filter to reduce the number of trainable parameters; the first temporal convolution layer is used to extract the feature map of the EEG signal in different frequency bands; the second temporal convolution layer is used to extract the feature map to represent data of different time scales; the self-attention layer is used to determine the most effective motor imagery information; the TCN layer is used to capture all available time information, and the extracted deep features are output after flattening and classified.

[0106] Specifically, Figure 4 As shown, the architecture of the deep part is described in detail, including traditional convolutional neural network (CNN), multi-head self-attention and temporal convolutional network. The deep network is inspired by the EEGNet series and further optimized by filter bank features and Transformer-based spatial features. Consider a single period of raw EEG data represented as R(C*T), and its corresponding label y{L, R, O}, where C is the number of EEG channels, T is the number of time points, and Nc is the number of different categories; L, R, O represent left leg, right leg and rest, respectively. Specifically, Nc=3.

[0107] Among them, EEGNet is a deep learning network architecture for processing electroencephalogram (EEG) data. Fig.10 As shown in Figure 1, it is a deep learning architecture specifically designed for processing EEG data, aiming to extract features from EEG data and perform tasks such as classification, recognition or regression. The design of this architecture takes into account the characteristics of EEG data, such as temporal, spatial and spectral information. It includes data preprocessing layers, convolution layers, pooling layers, flattening layers, fully connected layers and output layers. Lines represent the connectivity of convolution kernels between input and output (called feature maps). The network starts with temporal convolution (second column) to learn frequency filters, and then uses deep convolution (middle column) to connect to each feature map separately to learn frequency-specific spatial filters. Separable convolution (fourth column) is a combination of deep convolutions, which learns the temporal summary of each feature map separately, followed by a point convolution, which learns how to optimally mix these feature maps. The architecture of EEGNet can be adjusted and optimized according to specific tasks and data. It performs well in many EEG-related tasks, such as brain-computer interface, emotion recognition, disease diagnosis, etc. The success of this architecture is partly due to its ability to automatically learn features from raw EEG data without manually designing feature extractors.

[0108] Specifically, the deep processing network model of the EEG signal consists of five layers, namely, a deep convolution layer, a first temporal convolution layer, a second temporal convolution layer, a self-attention layer, and a TCN layer. The deep convolution layer extracts feature maps of the EEG signal in different frequency bands, and the first temporal convolution layer uses deep convolution to learn spatial filters and reduce the number of trainable parameters. The deep convolution layer is followed by a first average pooling layer to reduce the sampling rate. The second temporal convolution layer extracts feature maps to represent data at different time scales, and the second temporal convolution layer is followed by a second average pooling layer, the size of the second average pooling layer is P2, which is used to reduce the sampling rate. The fourth layer is a self-attention module, which helps the model focus on the most effective motor imagery information, and multi-head attention helps to focus on multiple positions and generate multiple attention representations. The fifth layer is a TCN network that further utilizes temporal information. The embodiment of the present invention stacks L residual blocks and selects the receptive field size to make RFS greater than 17, thereby allowing TCN to capture all available temporal information.

[0109] The outputs of all windows are concatenated and fed into the SoftMax classifier. This helps to increase data efficiently and improve classification accuracy.

[0110] The deep processing network model of EEG signals was trained and tested on a single GPU, using Nvidia GTX20708GB and the TensorFlow framework. The number of trainable parameters was 1000, and the training was performed for 3000 epochs. The training time was about 30 minutes, and the prediction time for a one-second EEG data segment was about 0.05 seconds. For all experiments, the embodiment of the present invention used the following training configuration. The weights were initialized using the Glorot uniform initializer, and the Adam optimizer with a learning rate of 0.0009 was used for training. The batch size was 64, and the categorical cross entropy loss was used for training within 1000 epochs. The stopping condition was 300. The hyperparameters are shown in Table 1. The hyperparameter settings apply to all subjects, whether in a specific subject scenario or an independent subject scenario. Among them, C = the number of EEG channels, T = the number of time samples, F1 = the number of temporal filters, F2 = the number of spatial filters, K1 = the convolution kernel size of the first convolution layer, K T = convolution kernel size of TCN module, F T = the number of filters in the TCN module. Here, T = 1125 time points, C = 22 EEG channels, Nc = 3 motor imagery categories, and I = 5184 motor imagery epochs.

[0111] Table 1

[0112]

[0113] Specifically, the self-attention layer adopts a multi-head attention mechanism, performs a single attention head in different spaces, maps Q, K and V to different spaces through linear transformation, concatenates the attention results of different spaces, performs a linear transformation and maps them to the output space to obtain the final result.

[0114] The steps of the whole process are as follows:

[0115] (1) Apply the projection matrix of each attention head to the inputs Q, K, and V respectively to obtain h different queries Q, keys K, and values ​​V; the parameter matrix of the projection matrix includes

[0116] (2) Perform self-attention calculation on each attention head to generate the output of h attention heads, namely head1 to head h , the formula is as follows:

[0117]

[0118] Where, i∈(1,h); and Represents the projection matrix of each attention head applied to the inputs Q, K, and V respectively;

[0119] (3) Concatenate the outputs of these attention heads together, which increases the dimension of the data; the concatenated result is a tensor with dimensions h*d_model; the concatenated result is combined with the weight matrix W O Multiply them together to perform a linear transformation and obtain the final output. The formula is as follows:

[0120] MultiHead(Q,K,V)=Concat(head1,...,head h )W O .

[0121] Among them, W O represents the weight matrix.

[0122] Specifically, the parameter matrix contains

[0123] represents the query projection matrix of the i-th attention head is a matrix over the real number field with dimension d model ×d k , where d model is the hidden layer dimension of the model, and d k is the dimension of the key vector; this projection matrix is ​​used to map the input query vector Q to a specific subspace for attention calculation;

[0124] represents the key projection matrix of the i-th attention head; is a matrix over the real number field with dimension d model ×d k , where d model is the hidden layer dimension of the model, and d k is the dimension of the key vector; this projection matrix is ​​used to map the input key vector K to a specific subspace for attention calculation;

[0125] represents the numerical projection matrix of the i-th attention head; is a matrix over the real number field with dimension d model ×d k , where d model is the hidden layer dimension of the model, and d k is the dimension of the key vector. This projection matrix is ​​used to map the input numerical vector V to a specific subspace for attention calculation;

[0126] Represents the output weight matrix W Ois a matrix over the real number field, with dimension hd v ×d model , where hd v is the dimension of the numeric vector, and d model is the hidden layer dimension of the model; this weight matrix is ​​used to map the output of the multi-head attention to the final model output dimension.

[0127] Specifically, the Transformer model is a model for machine translation proposed by Google in 2017. It completely abandons the structure of traditional recurrent neural networks and adopts a structure based entirely on the attention mechanism, which has achieved quite remarkable results. Since then, the full attention mechanism model design model has been expanded from the NLP field to the computer vision field. The Transformer model structure is as follows: Figure 5 As shown in the figure, it is divided into an encoder and a decoder, and the structures of the encoder and decoder are similar. The encoder is from bottom to top: the input first passes through the embedding layer, then the position encoding, and then passes through a residual connection and LayerNorm after passing through the multi-head attention, and finally passes through the feedforward neural network. The block composed of the position encoding, multi-head attention and feedforward neural network is called a Transformer block, and the encoder can be composed of several Transformer blocks. The decoder is the same. Since the Transformer adopts the self-attention mechanism, compared with the traditional RNN (Recurrent Neural Networks), LSTM (Long Short Term Memory), GRU (Gate Recurrent Unit) and other recurrent neural networks, there is no information about the order of positions, because the calculation of the attention mechanism is to calculate the attention weight of the current word and each character in the entire sentence, so even if the entire sentence is disrupted, the calculation result of the attention weight is the same. Therefore, in order to give it position information, the input information needs to be positionally encoded.

[0128] Due to its innovative self-attention mechanism and multi-head attention mechanism, Transformer has achieved revolutionary achievements in natural language processing, including models such as BERT (Bidirectional Encoder Representation from Transformers) and GPT (Generative Pre-Training). In addition to NLP (Natural Language Processing), Transformer has also been applied to other fields such as computer vision and speech recognition.

[0129] Specifically, ① Attention mechanism

[0130] The principle of the attention mechanism is as follows Figure 6 As shown, the weighted average is calculated using the input (query, q) and the output (key-value, kv) in the model, and the output matrix is:

[0131]

[0132] ②Multi-head attention mechanism

[0133] The principle of Multi-Head Attention is as follows Figure 7 As shown in Figure 2, it refers to the use of multiple attention heads simultaneously in the attention calculation process, and different attention weights learned from different projection subspaces. This helps the model capture information in the input sequence in different semantic spaces and provide a more comprehensive contextual understanding.

[0134] Specifically, all available temporal information is captured through the TCN layer, and the extracted deep features are output after flattening and classified, including:

[0135] Preprocess the EEG signals; for example, filtering and denoising, to reduce the impact of noise on classification performance.

[0136] Cutting the EEG signal into time windows and marking a corresponding movement intention category for each time window;

[0137] Construct a TCN model, wherein the TCN model consists of multiple temporal convolutional layers, each of which uses convolution kernels of different sizes to capture features of different time scales;

[0138] Apply global average pooling or global maximum pooling to capture the information of the entire sequence;

[0139] Connect the pooling output to the fully connected layer to perform the classification task and output predictions for different motion intent categories;

[0140] The TCN model is trained using the labeled training data, and the TCN model is optimized using a loss function and an optimization algorithm.

[0141] Specifically, Temporal convolutional network (TCN) refers to a temporal convolutional network, a new type of algorithm that can be used to solve time series prediction. The algorithm was first proposed by Lea et al. in 2016, when they were doing research on video action segmentation. Generally speaking, this conventional process includes two steps: first, low-level features are calculated using CNNs that (usually) encode spatiotemporal information, and second, these low-level features are input into a classifier that uses (usually) RNNs that capture high-level temporal information. The main disadvantage of this approach is that it requires two separate models.

[0142] The TCN model constructs a residual block to replace a layer of convolution. Figure 8 As shown in the figure, a residual block contains two layers of convolution and nonlinear mapping, and WeightNorm and Dropout are added to each layer to regularize the network.

[0143] Dilated causal convolution Fig. 9 As shown in the figure, the dilated convolution allows the input to be sampled at intervals during the convolution, and the sampling rate is controlled by d in the figure. d=1 in the bottom layer means that each point is sampled during the input process, and d=2 in the middle layer means that every 2 points are sampled as input during the input process. Generally speaking, the higher the level, the larger the value of d. Therefore, the dilated convolution makes the size of the effective window grow exponentially with the number of layers. In this way, the convolutional network can use fewer layers to obtain a large receptive field.

[0144] The advantage of TCN is that it can automatically learn time-related features from sequence data without the need for manual feature extraction, so it is suitable for EEG signal classification. The implementation process is to first preprocess the signal, preprocess the EEG signal, such as filtering and denoising, to reduce the impact of noise on classification performance. The second step is to cut and label the data, cut the EEG signal into appropriate time windows, and label the corresponding motion intention category for each time window. The third step is to build a TCN model, which consists of a series of temporal convolutional layers. Each convolutional layer uses convolution kernels of different sizes to capture features of different time scales. Convolutional layers may also include pooling layers or other nonlinear activation functions to extract and learn features. The fourth step is global average pooling: After the convolution layer, global average pooling or global maximum pooling can be applied to capture the information of the entire sequence. The fifth step is the classification layer, which connects the pooling output to a fully connected layer for performing classification tasks and outputs predictions for different action categories. The sixth step is training and optimization: Use labeled training data to train the TCN model, using appropriate loss functions and optimization algorithms. Finally, evaluation and testing are performed.

[0145] Among them, "DeepConvNet" and "ShallowConvNet" are both deep convolutional neural network architectures applied to EEG classification tasks. They represent deeper and shallower network structures, respectively, for extracting features from EEG signals and performing classification.

[0146] "DeepConvNet" refers to a deep convolutional neural network, which was first proposed by Schirrmeister et al. in a paper. DeepConvNet usually contains multiple layers of convolutional layers and pooling layers, as well as some fully connected layers and activation functions. This architecture uses convolutional layers to capture local features of sequence data, while pooling layers can reduce data dimensions while retaining key information. The application of DeepConvNet in the field of brain-computer interface is mainly for EEG signal classification, such as classifying different movement intentions into different categories.

[0147] "ShallowConvNet" is a relatively shallow convolutional neural network architecture proposed by Schirrmeister et al. in a paper. It uses fewer convolutional layers and parameters than DeepConvNet, and focuses on the local spatiotemporal features of EEG signals. This network structure is designed to improve computational efficiency while maintaining reasonable classification performance. ShallowConvNet is also used in EEG signal classification tasks, such as classifying different motion intentions.

[0148] In summary, DeepConvNet and ShallowConvNet are both deep convolutional neural network architectures designed to handle EEG signal classification tasks. They differ in network depth and structure, but both aim to extract useful time-frequency features from EEG signals for classification. The choice of which architecture depends on task requirements, computing resources, and performance requirements.

[0149] Specifically, it also includes: using LOSO cross-validation evaluation technology to evaluate performance;

[0150] Using accuracy and Cohen's Kappa coefficient as performance indicators, as shown in formula (1) and formula (2), the performance of the algorithm is evaluated by statistically analyzing the Cohen's Kappa coefficient k;

[0151]

[0152] Among them, TP represents a true positive example of a certain class, FP represents a false positive example, FN represents a false negative example, and TN represents a true negative example;

[0153]

[0154] Among them, P oRepresents the classification accuracy, P e Represents the results of random classification.

[0155] Specifically, the method proposed in the embodiment of the present invention has been investigated by two evaluation methods: subject-specific and subject-independent. In the subject-specific analysis, the embodiment of the present invention uses the first sequence data of each subject to train the model, and tests it on the data of the second and third sequences of the same subject. As for the individual independent evaluation, it is usually called "cross-subject validation", or "cross-subject evaluation", that is, the evaluation method of "leaving one subject out" (LOSO), which is used for each sequence. The model is trained and evaluated through several folds, and the number of folds is equal to the number of subjects. For each fold, one subject is used for evaluation, and the other subjects are used for training. The LOSO evaluation technique ensures that the model is evaluated on different subjects that do not appear in the training data. It is worth noting that in LOSO, it is necessary to avoid using cross-sequence data to prevent mixed noise across sequences.

[0156] Specifically, the performance evaluation of the method proposed in the embodiment of the present invention uses accuracy and Cohen's Kappa coefficient as performance indicators, as shown in formula (1) and formula (2). In addition, the embodiment of the present invention also performs a statistical analysis on the Cohen's Kappa coefficient k to evaluate the performance of the algorithm.

[0157]

[0158] Among them, TP represents a true positive example (True Positive) of a certain class, FP represents a false positive example (False Positive), FN represents a false negative example (False Negative), and TN represents a true negative example (True Negative).

[0159]

[0160] Among them, P o Represents the classification accuracy, P e Represents the results of random classification.

[0161] First, the embodiment of the present invention evaluated the learning effect to determine whether there is a significant performance improvement when the training sequence is increased. In addition, the embodiment of the present invention performed feature analysis to evaluate the possible differences in the discriminative features between the two feedback modalities. In order to estimate the frequency band range effect of the EEG, the embodiment of the present invention calculated the scalp distribution of energy by averaging the discriminative energy of different frequency bands in the two sequences.

[0162] Next, the proposed method is compared with a classic machine learning algorithm FBCSP-LDA and three advanced convolutional neural network architectures, namely EEGNet, EEG-TCNet and ATCNet. Since the method proposed in the embodiment of the present invention includes two parts, shallow and deep, a elimination analysis is performed to measure the effectiveness of each algorithm module.

[0163] Finally, binary classification of motor imagery and resting (baseline) EEG signals was performed, and the learning effect was evaluated based on the results of binary classification. The features (channel and frequency pairs) that distinguish motor imagery and resting under the two feedback conditions were obtained with the help of classifiers. An optimized classifier was constructed for each subject in each sequence to evaluate these features and EEG modalities. In addition, paired sample t-tests were performed to evaluate significant differences in online decoding performance. The significance level was set to 0.05. When there were significant differences, the Tukey correction method was used to correct statistical errors through post-hoc multiple comparisons. This correction method helps to control the false alarm rate that may occur when multiple comparisons are made and ensure the statistical credibility of the conclusions.

[0164] Specifically, as shown in Table 2, the classification accuracy (%) and kappa values ​​of all subjects in the specific subject and independent subject situations, Table 2 shows the real-time brain-computer interface (BMI) performance of the participants in the online sequence. The average accuracy of all subjects was 56.84±4.14% in the visual feedback sequence and 60.27±3.51% in the proprioceptive feedback sequence. Among them, the accuracy of proprioceptive feedback was significantly higher than that of the visual feedback sequence (one-tailed two-sample t-test, p<0.05, Bonferroni correction).

[0165] In addition, the average accuracy of all subjects was 53.97 ± 3.07% in LOSO1, 54.54 ± 1.79% in LOSO2, and 55.39 ± 2.49% in LOSO3. There was no statistical difference in the accuracy between the various LOSO methods (two-sample t-test, p > 0.05, Bonferroni correction).

[0166] In addition, the embodiment of the present invention calculated the random level based on the three-category classification. The probability of each category being guessed as the third category is 0.33, so the accuracy of the random level is 33.3%, and a random accuracy of 33.3% is obtained. The classification accuracy of sequence V and sequence P is significantly higher than the random level (one-tailed two-sample t-test, p < 0.05, Bonferroni correction). The performance of the LOSO session is also significantly higher than the random level (one-tailed two-sample t-test, p < 0.05, Bonferroni correction).

[0167] Table 2

[0168]

[0169] As shown in Table 3, for the three-class classification accuracy and kappa values ​​(3 categories, i.e., left, right, and resting) reproduced by the proposed model and other methods, these indicators were averaged for all subjects in each specific subject and independent subject context, respectively. Table 3 summarizes the accuracy and K value of the proposed model, compared with the reproduced EEGNet, EEG-TCNet, and ATCNet. The results of the reproduced model are based on the hyperparameters given in the original article, while the preprocessing, training, and evaluation followed the steps proposed in the embodiments of the present invention. The performance of all subjects was better than EEGNet, EEG-TCNet, and ATCNet, with an improvement of up to 4% and 2% in accuracy in individual-specific and individual-independent modes, respectively. The performance of all methods was also significantly higher than the random level (one-tailed two-sample t-test, p < 0.05, Bonferroni correction). In addition, the average confusion matrix of ATCNet and the reproduced model is shown in Figure 3. Fig.11 As shown in Figure 2, ATCNet performs better than other models in decoding all MI categories.

[0170] Table 3

[0171]

[0172] Table 4 shows the accuracy and kappa scores of the two-category classification (2 classes, i.e., imagery and rest) reproduced for the proposed model and other methods. These metrics were averaged for all subjects in each specific subject and independent subject context. Table 4 summarizes the accuracy and kappa scores of the proposed model compared with the reproduced EEGNet, EEG-TCNet, FBCSP, and ATCNet for the two-class motion classification of rest and motion. Across all subjects, the average accuracy was 55.42% for the visual feedback and 64.20% for the proprioceptive feedback sequences. The chance level was calculated based on the binomial distribution, resulting in a chance accuracy of 50%. The classification accuracy of both the visual feedback and proprioceptive feedback experimental sequences was significantly higher than the chance level (one-tailed two-sample t-test, p<0.05, Bonferroni correction). The performance shown by the leave-one-out sequence, i.e., accuracy and kappa score, was also significantly higher than the chance level (one-tailed two-sample t-test, p<0.05, Bonferroni correction). As shown in Figure 12(a) and Figure 12(b), the average EEG signals of the Cz channel in the two feedback modes change over time.

[0173] Table 4

[0174]

[0175] In order to realize a natural, intuitive, and user-friendly lower limb exoskeleton brain-machine interface (BMI) system, a comprehensive understanding of the neural mechanisms of lower limb movement is necessary. Neural features can be decoded from movement execution or motor imagery. Movement-related cortical potentials (MRCPs) are commonly used to detect motor intentions in motor planning and execution, which can be detected about 1.5 to 2 seconds before the actual movement begins. The existing technology detects the movement execution of ankle dorsiflexion by using manifold method and linear discriminant analysis (LDA) as classifiers, and further studies the influencing factors of lower limb movement intention, including movement type (dorsiflexion or plantar flexion), limb side (left leg or right leg), and processing method (time series analysis based on MRCP, i.e., movement-related cortical potentials, or frequency domain estimation based on SMR, i.e., sensorimotor rhythm), frequency band (e.g., delta, theta, mu, beta, and MRCP band at [0.11] Hz). During the conversion from offline to online BMI, detection delay is a common problem for this EEG modality, mainly due to the use of causal spectral filters, which usually cause large delays in the system. In rehabilitation systems, short delays are very important to improve the effect of neural repair.

[0176] Motor imagery (MI) activity is often used to control brain-computer interface (BMI) systems. Motor imagery is generated when imagining performing an action, but it can also be observed when actually performing the action. Movement-related cortical potential (MRCP) activity occurs in the frequency band of 0-5Hz, while motor imagery activity is observed between 8-30Hz. These frequencies fall in the delta (<4Hz), theta (4-7Hz), mu (8-12Hz), and beta (13-30Hz) frequency bands, respectively. Therefore, for a specific user, it is necessary to first evaluate whether the user's brain activity frequency band is most intense in the MRCP frequency or in the MI frequency, which means that each time the user's intention is identified, the frequency band, i.e., the feature, needs to be selected based on the user's performance.

[0177] For the control of rehabilitation exoskeleton robots, the most intuitive and natural method is to identify the EEG intention of the lower limb exoskeleton (i.e., imagine the movements of the left and right legs) and use it to control the lower limb exoskeleton device. Motor imagery is a cognitive activity that reactivates any first-person motor performance internally to perform a virtual rehearsal of the actual movement. In this process, individuals reproduce and simulate various aspects of the actual movement, including the details, sequence, and movement path of the movement execution, through internal neural activation and brain simulation without producing explicit motor output. This mechanism of cognitive rehearsal enables individuals to simulate and reproduce motor behavior in the brain, thereby achieving mental training and preparation for motor planning, coordination, and skills without the need for actual muscle movement. In addition, in the cerebral cortex, the neuronal clusters specific to lower limb movement are relatively limited in space. This may be because lower limb movement does not involve complex neural networks and coordinated motor control mechanisms, so that the neural areas specific to each leg do not occupy a large number of neurons. Therefore, it is difficult to identify and capture specific lower limb movement intentions in a relatively small brain area to achieve more accurate movement control and guidance. It has been proven in the prior art that it is possible to distinguish between imagined left and right lower limb movements, with a classification accuracy of 63%. The main information comes from the alpha and beta frequency bands of electrodes C1, C2, PO3 and PO4, and the accuracy obtained in the embodiments of the present invention is within this recognition range. The introduction of more advanced methods may lead to further performance improvements. Other lower limb experimental paradigms include pre-gait movement, gait cycle and movement coordination. Decoding kinematic parameters such as velocity information from lower limb joint movements can achieve continuous trajectory reconstruction.

[0178] Another important topic involves lower limb powered robotics. Repeated rehabilitation training is essential for the brain's ability to reshape nerves and adapt to new situations. The lower limb exoskeleton device proposed in the embodiment of the present invention has been proven to have significant effects in improving the walking ability, balance and muscle strength of individuals with lower limb injuries or disabilities.

[0179] Finally, individual differences in electroencephalogram (EEG) data affect the reusability and generalization ability of the model. Within the same subject, the performance of the model is significantly better than the LOSO method (leave one out of the group). Transfer learning can adjust the model to be more suitable for small-scale data sets while maintaining the ability to learn individual differences. Commonly used transfer learning methods in EEG signal analysis include domain adaptation, improved CSP algorithm, deep neural network (DNN) algorithm, and subspace learning. It should be noted that in addition to being applied to processing raw EEG data, deep learning can also be further optimized to achieve artifact detection and removal of more neural signals.

[0180] Specifically, an integrated lower limb rehabilitation exoskeleton EEG decoding system includes:

[0181] EEG equipment, used to obtain EEG signals from the user's motor cortex;

[0182] A first processing module is used to perform shallow processing on the EEG signal to obtain a first type of feature, i.e., shallow feature;

[0183] The second processing module is used to construct a deep processing network model of the EEG signal, perform deep processing on the EEG signal, and obtain the second type of features, i.e., deep features;

[0184] A decoding module is used to concatenate the first type of features and the second type of features, and send them to a trained classifier (the first sequence of EEG signals is used to train the classifier) ​​to decode the movement intention of the EEG signal;

[0185] The feedback module is used to provide visual feedback to the user based on the decoded movement intention, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

[0186] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0187] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for EEG decoding of lower limb rehabilitation exoskeleton based on ensemble learning, characterized in that: The following steps are involved: Obtain EEG signals from the user's motor cortex through an electroencephalogram device; The EEG signal is shallowly processed to obtain the first type of features; Construct a deep processing network model of EEG signals, perform deep processing on EEG signals, and obtain the second type of features; The first and second category features are combined, and a classifier is trained to decode the movement intention of the EEG signal; Provide visual feedback to the user based on the decoded movement intention, and then execute the decoded movement on the lower limb exoskeleton device to obtain proprioceptive feedback; Among them, the deep processing network model includes a TCN layer, which is used to automatically learn time-related features from sequence data, capture all available time information, output the extracted deep features after flattening and classify them; the TCN layer is a temporal convolutional network, which replaces one layer of convolution with a residual block. A residual block contains two layers of convolution and nonlinear mapping, and WeightNorm and Dropout are added to each layer to regularize the network.

2. According to the method of claim 1, the EEG decoding method of the rehabilitation lower limb exoskeleton based on ensemble learning is characterized in that: Shallow processing of EEG signals includes: Filter and wavelet transform the EEG signal to obtain different frequency components of the EEG signal; The different frequency components are processed by cospatial pattern analysis to separate the EEG signal into additional subcomponents with the largest variance differences between the different categories; The features processed by co-spatial pattern analysis in different frequency components are combined to obtain the first type of features.

3. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 1, characterized in that: The EEG signal of the user's motor cortex includes: offline sequence data, first online sequence data, and second online sequence data; The deep processing network model of EEG signals is trained through offline sequence data; The deep processing network model of the EEG signal is tested by the first online sequence data and the second online sequence data.

4. The method for EEG decoding of a lower limb exoskeleton based on ensemble learning according to claim 1, characterized in that: The deep processing network model of EEG signals also includes: deep convolution layer, first temporal convolution layer, second temporal convolution layer, and self-attention layer; Using deep convolution to learn spatial filters through deep convolutional layers reduces the number of trainable parameters; Extract feature maps of EEG signals in different frequency bands through the first temporal convolutional layer; The feature maps are extracted through the second temporal convolution layer to represent data at different time scales; Determining the most effective motor imagery information via a self-attention layer.

5. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 4 is characterized in that: The self-attention layer adopts a multi-head attention mechanism, which performs a single attention head in different spaces, maps Q, K and V to different spaces through linear transformation, concatenates the attention results of different spaces, performs a linear transformation and maps them to the output space to obtain the final result.

6. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 5, characterized in that: The process of concatenating the attention results of different spaces and performing a linear transformation to map them to the output space is as follows: Apply the projection matrix of each attention head to the input Q, K and V respectively to obtain h different queries Q, keys K and values ​​V; the parameter matrix of the projection matrix includes Perform self-attention calculation on each attention head to produce the output of h attention heads, the formula is as follows: Where, i∈(1,h); and Represents the projection matrix of each attention head applied to the inputs Q, K, and V respectively; The outputs of these attention heads are concatenated together and weighted to perform a linear transformation to obtain the final output. The formula is as follows: MultiHead(Q,K,V)=Concat(head1,…,head h )W O ; Among them, W O represents the weight matrix.

7. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 4, characterized in that: All available temporal information is captured through the TCN layer, and the extracted deep features are output after flattening and classified, including: Preprocess the EEG signal; Cut the EEG signal into time windows and label each time window with the corresponding movement intention category; Construct a TCN model, wherein the TCN model consists of multiple temporal convolutional layers, each of which uses convolution kernels of different sizes to capture features of different time scales; Apply global average pooling or global maximum pooling to capture the information of the entire sequence; Connect the pooling output to the fully connected layer to perform the classification task and output predictions for different motion intent categories; Use labeled training data to train the TCN model, and optimize the TCN model through loss function and optimization algorithm.

8. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 3, characterized in that: The acquisition of offline sequence data includes: Preset the first prompt time, rest time period, visual prompt time period, motor imagery time period and second prompt time; The user is prompted through the display screen. The user starts from the first prompt time, goes through a rest period, and then indicates the movement direction through visual prompts. The user performs offline training of motor imagery according to the visual prompts. After the motor imagery period ends, a second prompt is given; the acquisition of offline sequence data for one trial ends; The first online sequence data is based on visual feedback, and an online test of visual feedback is performed during the motor imagery period to obtain discrete visual feedback; The second online sequence data is based on proprioceptive feedback, with visual feedback during the motor imagery period, and then the decoded movement is executed on the lower limb exoskeleton device to obtain proprioceptive feedback.

9. The method for EEG decoding of a lower limb rehabilitation exoskeleton based on ensemble learning according to claim 1, characterized in that: Also includes: The LOSO cross-validation evaluation technique was used for performance evaluation; Using accuracy and Cohen's Kappa coefficient as performance indicators, as shown in formula (1) and formula (2), the performance of the algorithm is evaluated by statistically analyzing the Cohen's Kappa coefficient k; Among them, TP represents a true positive example of a certain class, FP represents a false positive example, FN represents a false negative example, and TN represents a true negative example; Among them, P o Represents the classification accuracy, P e Represents the results of random classification.

Images