Incremental learning method for motor imagery task based on stacked width learning system

Abstract

The application discloses a motor imagery task incremental learning method based on a stacked width learning system, band-pass filtering is performed on motor imagery electroencephalogram of an initial task, time-frequency decomposition is performed on the filtered signal by using wavelet packet transformation, and a plurality of narrow-band signals are obtained after signal reconstruction; global spatial features are extracted based on sub-band optimization and common spatial pattern algorithm, local spatial features are extracted through Riemann manifold embedding and feature optimization; for the initial motor imagery task, the global and local spatial features are simultaneously input into a double-branch feature mapping layer to establish a first sub-width learning system, and system parameters are kept frozen and shared in the incremental task stage; for each incremental task, a new sub-system is stacked in a residual connection mode to maintain the memory of the old task, and a new enhanced node is used to learn the characteristics of the new task. The method has been verified on a public data set, and the data set contains six types of motor imagery tasks of three different parts of the upper limbs.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface technology and is designed as an incremental learning method for upper limb motor imagery tasks based on a stacked width learning system. Specifically, it involves: extracting global and local spatial features from the brain electronic frequency band of motor imagery; stacking multiple sub-width learning systems with output layer weight importance constraints in a residual connection manner to efficiently identify new motor imagery tasks and retain the memory of old tasks. Background Technology

[0002] In recent years, the high incidence and disability rate of stroke have seriously threatened human life and health. As an emerging scientific field that intersects brain science, information science, and artificial intelligence, brain-computer interface (BCI) technology has provided an effective approach for limb neurological rehabilitation after stroke. Motor imagery (MI) BCI, as a non-invasive method, allows the computer to decode the movement intention of the affected limb and control external devices to assist limb movement simply by imagining the movement without actual motor output. Motor function rehabilitation often progresses from simple movements of a single limb to complex movements involving multiple limbs. Currently, most publicly available MI decoding methods can effectively decode multiple tasks acquired simultaneously. However, when faced with an increasing number of tasks, due to significant differences in activated brain regions and signal intensity between tasks, the model updated by the new task exhibits "catastrophic forgetting" of previous tasks, especially evident in deep neural network models. Therefore, decoding methods for incremental MI tasks are more practically significant and have greater application value.

[0003] By combining the advantages of stochastic vector function chain networks and feedforward neural networks, the Broad Learning System (BLS) provides an efficient and flexible wide-scalable incremental model, exhibiting significant advantages under the setting of independent and identically distributed incremental features. However, for tasks with complex and varied feature space distributions, especially MI tasks based on high-dimensional, time-varying Motor Imagery Electroencephalography (MI-EEG) signals, BLS often struggles to accurately establish the correlation between the activation intensity of different channels and the MI task. On the other hand, due to the low sensitivity of model parameters caused by randomly determining connection weights, BLS cannot adapt to new MI tasks while maintaining memory of old tasks.

[0004] To address the aforementioned issues, this invention proposes an incremental learning method for upper limb motor imagery tasks based on a stacked width learning system, and trains and validates this method using a publicly available dataset. This method integrates global spatial features based on an Adaptive Multi-band Common Spatial Pattern (AMCSP) and local spatial features based on a Riemannian manifold (RM) to form a multi-view feature matrix describing the spatial feature distribution of different MI tasks. Furthermore, multiple stacked sub-BLSs are residually connected to the BLS trained on the initial task to decode new tasks, and the importance of output layer weights is constrained to the new sub-BLSs to mitigate the model's forgetting of old tasks. The method disclosed in this invention aims to continuously decode incremental MI tasks. Summary of the Invention

[0005] This invention aims to propose a universal method for incremental learning of upper limb motor imagery tasks, thereby enhancing the practical application value of brain-computer interface motor imagery paradigms. Specific invention details include:

[0006] (1) Extract global and local spatial features respectively. Design multi-layer wavelet packet transform to perform time-frequency decomposition on the original signal, and use inverse wavelet transform to reconstruct it into multiple signals. Select the optimal frequency band according to the energy value of the decomposed sub-frequency band, and extract global spatial features through CSP; use RM to extract local spatial features of the reconstructed signal from all sub-frequency bands, and select key features.

[0007] (2) Establish BLS-1 for recognition task 1. Input the global and local spatial feature matrices of task 1 to different groups of feature mapping nodes of BLS, generate mapping features and enhancement features, calculate the connection weights of the output layer, and expand the model to convergence by adding feature mapping nodes and enhancement nodes.

[0008] (3) Establish BLS-n to identify n incremental tasks. Fix the parameters of the old task model, and stack a sub-BLS for each new task in the form of residual connections. The new sub-BLS inherits some parameters of the old model and introduces the importance constraint of the output layer weights, while providing new enhancement nodes to learn new tasks.

[0009] The specific steps involved are as follows:

[0010] Step 1: Signal preprocessing and time-frequency decomposition.

[0011] Step 1.1 Define the input signal. Define the input dataset as follows: ,in , T represents the number of samples, S represents the number of time points in a single sample, C represents the number of lead channels, and L represents the category label.

[0012] Step 1.2 Signal Preprocessing. Design an 8th-order Butterworth bandpass filter with a bandpass range of 4-30Hz. Expand the dataset using the sliding window method.

[0013] Step 1.3 Time-frequency decomposition and sub-band signal reconstruction. The filtered signal from Step 1.2 is decomposed into multiple sub-band signals using wavelet packet transform, and the sub-band signals are reconstructed using inverse wavelet transform.

[0014] Step 2: Extract global and local spatial features.

[0015] Step 2.1 Extract global spatial features. Calculate the energy value of each sub-band in Step 1.3. Starting from the band with the highest energy value, adaptively select the optimal bands with energy values ​​greater than 0.9, alternating between higher and lower bands. Reconstruct the filtered signal of each selected band. Introduce the CSP algorithm to calculate the global spatial features of each band and form a feature matrix. .

[0016] Step 2.2 Extract local spatial features. Reconstruct the signals of all sub-bands from Step 1.3, forming a two-dimensional signal matrix with the number of sampling time points on the horizontal axis and the number of electrode channels (ch) on the vertical axis. Construct the matrix into a symmetric positive definite matrix and map it onto a Riemannian manifold. The matrix dimension is... The Riemann mean is calculated as the reference center point of the tangent space. The Riemann symmetric positive definite matrix of each frequency band is projected into the tangent space to obtain the tangent space features in Euclidean space. Features are selected through supervised feature sparsification to obtain the local spatial feature matrix. .

[0017] Step 3: Train BLS-1 for Task 1.

[0018] Step 3.1 Multi-view feature fusion. Input the feature matrices from Step 2.1 and Step 2.2 into the dual-branch feature mapping layer to obtain the mapping features for Task 1. ,in This represents the number of windows for the feature mapping nodes.

[0019] Step 3.2 Identify Task 1. Map the features. Input feature enhancement layer to obtain enhanced features Finally, Step 3.1 and Convert to The connection weights of the output layer nodes are calculated using ridge regression. Thus, input features are established. , With model output The mapping relationship is used to obtain the sub-BLS of recognition task 1, denoted as BLS-1.

[0020] Step 4: Evaluate the importance of the output weights and encapsulate the model.

[0021] Step 4.1 Evaluate the output layer connection weights Importance. Output layer connection weights are calculated using Fisher information matrix and Bayesian inference. Importance of Task 1.

[0022] Step 4.2 Encapsulate the model of Task 1, denoted as TiBLS-1 (Task Increment broad learningsystem -1, and the abbreviations thereafter are similar). TiBLS-1 includes the feature construction methods provided in Step 1 and Step 2 and the mapping relationship of BLS-1 in Step 3.

[0023] Step 5. Identify n new MI tasks.

[0024] Step 5.1 Feature Representation. Repeat Step 1-Step 2 to obtain two feature matrices for the current task.

[0025] Step 5.2 Add a sub-BLS. Call the model BLS-(n-1) of the previous n-1 tasks, which contains n-1 stacked sub-BLSs, and calculate the mapping features. and enhanced features The connection weights and biases are randomly generated in Task 1 and set as shared parameters in the incremental tasks, where they are no longer updated. Calculate the residuals of Task n in the BLS-(n-1) model.

[0026] Step 5.3 Update the output weights of the newly added sub-BLS. Based on the residuals in Step 5.2, and combined with the output layer connection weights and weight importance matrices of the previous (n-1) tasks, calculate the output layer connection weights of the BLS-n of the nth task, and stack this sub-BLS to the depth direction of the model BLS-(n-1) of the previous task in a residual connection manner.

[0027] Step 5.4 Adding Enhancement Nodes. In the feature enhancement layer of the sub-BLS in Step 5.3, new nodes are added in the width direction to learn the nonlinear features of the new task and improve the convergence of the new sub-BLS. The newly added enhancement nodes are not shared between subsystems.

[0028] Step 5.5 Repeat Step 4, and encapsulate the model as TiBLS-n. Attached Figure Description

[0029] Figure 1 The method flowchart of the present invention.

[0030] Figure 2 Signal acquisition timing diagram.

[0031] Figure 3 A schematic diagram of a multi-layer sub-BLS stacked network structure. Detailed Implementation Plan

[0032] To illustrate the purpose, technical solution, and advantages of the method of the present invention in detail, the specific implementation scheme is described below with reference to the accompanying drawings:

[0033] This invention proposes an incremental learning method for upper limb motor imagery tasks based on a stacking width learning system, used for continuous learning of different motor imagery tasks. The flowchart of the invention method is shown below. Figure 1 As shown. To verify the feasibility of this invention, the model in the proposed method was trained using the publicly available upper limb motion dataset: BNCI Horizon 2020 (BNCI for short), and the test accuracy was obtained. The dataset is described as follows:

[0034] The BNCI dataset provides EEG signals from 15 healthy subjects (mean age 27 years) in two categories: motor execution and motor imagery. These included seven tasks: elbow flexion and extension, wrist internal and external rotation, hand opening and closing movements, and resting state. Signals were acquired using a 64-lead electrode cap conforming to the international 10-20 standard, with 61 electrode positions recording EEG signals at a sampling frequency of 512 Hz. A time-series diagram of the signal acquisition experimental paradigm is shown below. Figure 2 As shown. At 0 seconds, a "+" sign appears in the center of the display accompanied by a brief beep, prompting the subject to focus their attention; at 2 seconds, a task instruction appears on the display; from 2 to 5 seconds, the subject performs the corresponding task's motor execution or visualization. Motor execution and motor visualization were each tested in 10 rounds, with 42 EEG signals recorded per round (including 7 types of tasks, 6 times per type). This invention used 6 types of motor visualization (excluding resting state) EEG signals for verification, gradually incorporating them in the order of right upper limb elbow flexion and extension (Task 1), hand opening and closing (Task 2), and wrist internal and external rotation (Task 3) to verify the continuous learning ability of the proposed method. Specific implementation is as follows:

[0035] Step 1.1 For motor imagery tasks 1: right upper limb elbow flexion and extension, task 2: right upper limb hand opening / closing, and task 3: right upper limb wrist internal / external rotation, define the input dataset as follows: ,in , T represents 120 samples, S represents 1536 sampling points, C represents 61 leads, and L represents two types of tasks.

[0036] Step 1.2.1 The sliding window method was used to expand the dataset, with a time window length of 1 second and a step size of 0.2 seconds. The number of tasks in each category was expanded to 1080.

[0037] Step 1.2.2 Filter the signal expanded in Step 1.2.1 using an 8th-order Butterworth bandpass filter, setting the bandpass range to 4-30Hz, and extract the rhythm most relevant to the motion imagery task.

[0038] Step 1.3 uses a 4-layer wavelet packet transform to decompose the preprocessed signal obtained in Step 1.2 into 16 sub-bands with a bandwidth of approximately 2Hz. The wavelet basis function is Daubechies, and the filter length is 4. Each sub-band signal is then reconstructed using an inverse wavelet transform.

[0039] Step 2.1.1 Calculate the energy value of each sub-band in Step 1.3, as shown in Formula (1).

[0040] (1)

[0041] in This is the sub-band number. For sampling points, For the first The first sub-band Wavelet packet coefficients at each sampling point. Compare the energy values ​​of sub-bands, starting with the sub-band with the highest energy value. Initially, the optimal frequency bands with energy values ​​greater than 0.9 are adaptively selected alternately in the directions of increasing and decreasing frequency, forming an optimal frequency band set. ,

[0042] (2)

[0043] in Control the direction of sub-band selection, set , It is the number of the preferred frequency bands.

[0044] Step 2.1.2 Introduce the CSP algorithm to calculate the global spatial characteristics of the preferred frequency band signal in Step 2.1.1. The optimal number of spatial filter banks is chosen to be 2.

[0045] Step 2.2.1 For the reconstructed signals of all sub-bands in Step 1.3, construct the covariance matrix of the reconstructed signals, i.e. ,in According to the principle of Riemannian manifolds, each covariance can be considered as a point in the manifold; therefore, the distance between any two points is... As shown in formula (3),

[0046] (3)

[0047] in If it is the Frobenius norm, then the Riemann mean is expressed as:

[0048] (4)

[0049] in Using the reference center point of the tangent space, the points in the manifold are projected into the tangent space, and the upper triangular elements of the tangent space matrix are extracted as the tangent space features, as shown in formula (5).

[0050] (5)

[0051] in , Let be the reference matrix of the Riemannian manifold, and... The relationship is , representing the local homomorphism of Euclidean tangent space and Riemannian manifold space.

[0052] Step 2.2.2 For the tangent space feature vectors in Step 2.2.1, supervised feature sparsification is used to select features, including high-dimensional features and feature labels. Define the low-dimensional feature vector as The transformation vector is Then the feature optimization objective is expressed as:

[0053] (6)

[0054] but ,pass The size of the value is used to measure the importance of each feature dimension. The regularization coefficient is selected from the previous one. Dimensional features are denoted as .

[0055] Step 3.1.1 Global spatial features from Step 2.1 and Step 2.2 and local spatial features Each is mapped to a different feature space, such as Figure 3As shown, the mapping relationship is expressed as in formulas (7) and (8).

[0056] (7)

[0057] (8)

[0058] in For a linear mapping function, the connection weights of the k-th window are... , and bias , Randomly generated features are used, and a sparse autoencoder approach is introduced to optimize the mapping features.

[0059] Step 3.1.2 Obtain the mapping features based on Step 3.1.1 .

[0060] Step 3.2.1 Mapping features from Step 3.1.2 The input feature enhancement layer, the mapping relationship is represented as:

[0061] (9)

[0062] in ReLU is a non-linear mapping function, where the feature mapping layer contains Each node, connection weights of the feature enhancement layer and paranoia Randomly generated.

[0063] Step 3.2.2 Combine Step 3.1 and spliced ​​as The input is then fed into the output layer, and the connection weights of the output layer are calculated using ridge regression. Thus, input features are established. With model output The mapping relationship is used to obtain the sub-BLS of recognition task 1, denoted as BLS-1.

[0064] Step 4.1 Calculate the importance of each output layer connection weight to Task 1 using the Fisher information matrix. The calculation formula is shown in formula (10), where The number of samples for Task 1:

[0065] (10)

[0066] Step 4.2 Encapsulate the task incremental learning model TiBLS-1 for Task 1, where TiBLS-1 includes the feature construction methods provided in Step 1 and Step 2 and the mapping relationship of BLS-1 in Step 3.

[0067] Step 5.1 Repeat Step 1-Step 2 to obtain the global features of the newly added task n. and local features , where n=2, 3, and will be referred to as n in the following text.

[0068] Step 5.2 Call the BLS-(n-1) model of the first n-1 tasks and calculate the residual of task n in the previous task models. ,in This represents the model output in the sub-BLS corresponding to task n in the i-th task, with the residual linking method as follows: Figure 3 As shown.

[0069] Step 5.3.1 Based on the residuals in Step 5.2, combine the output layer connection weights and weight importance matrices of the first n-1 sub-BLS. The objective function of the nth layer BLS is defined as:

[0070] (11)

[0071] in Spatial mapping features and augmentation features for the nth task This is obtained from Step 1 to Step 3. The output layer weights of the sub-BLS corresponding to the nth task are... For categories of motion imagination tasks, It is a balancing coefficient for the importance of tasks. It is the element-wise Hadamard product. The change in the output layer weights of the sub-BLS corresponding to the nth task is the same as the change in the output layer weights of the sub-BLS corresponding to the i-th task.

[0072] Step 5.3.2 By diagonalizing the block matrix, take the partial derivative of formula (11) and set it to 0 to obtain... The expression:

[0073] (12)

[0074] in The output weights for the c-th category are... for The diagonalized matrix, , , , As defined in Step 5.3.1, stack BLS-n on top of the previous BLS-(n-1) layers.

[0075] Step 5.4 In the feature enhancement layer of sub-BLS-n in Step 5.3, enhancement nodes are adaptively added to improve the convergence of the model on the new task. The mapping relationship of the newly added nodes is represented as follows: , and These represent the connection weights and biases of the newly added nodes. For spatial mapping features, For linear mapping, This represents the enhanced features. The features after adding enhanced nodes are represented as follows: , This refers to the spatial mapping features and enhanced features before adding nodes. The newly added enhanced nodes are not shared between subsystems.

[0076] Step 5.5 Repeat Step 4, encapsulate the model as TiBLS-n, and represent the model output as follows: , This represents the output of task n in each sub-BLS layer.

[0077] Step 5.6 The proposed method was validated using the BNCI dataset. The recognition accuracy for the three incremental tasks is shown in Table 1 below. Task 1 is the flexion and extension of the right upper limb elbow, Task 2 is the opening / closing of the right upper limb hand, and Task 3 is the internal / external rotation of the right upper limb wrist. According to Table 1, the proposed method achieved a recognition accuracy of 70.08% in the 6-class classification tasks of the three incremental tasks, and did not show a significant performance degradation on the previous tasks. The experiment verified the reliability of the proposed method.

[0078] Table 1. Recognition accuracy of the method of the present invention for new and previous tasks.

[0079]

Claims

1. An incremental learning method for motion imagery tasks based on a stacked width learning system, characterized in that, Includes the following steps: Step 1: Signal preprocessing, time-frequency decomposition, and sub-band signal reconstruction; the signal is a motor imagery EEG signal. Step 2: Design a global spatial feature extraction algorithm based on sub-band optimization and common space mode, and a local spatial feature extraction algorithm based on Riemannian manifold embedding and feature optimization, to extract the global and local spatial features of the reconstructed signal in Step 1, respectively. Step 3: Integrate the features extracted in Step 2 into global and local multi-view space features, and use them as input to train the task-oriented sub-BLS, denoted as BLS-1; Step 4: Evaluate the importance of the BLS-1 output weights in Step 3 to Task 1, encapsulate the model, and denote it as TiBLS-1; Step 5: For the new task, repeat Steps 1 to 4 and introduce the importance constraint of the output layer weight to guide the update of the new sub-BLS parameters. Stack multiple sub-BLS corresponding to the task in the form of residual connections. Step 5 specifically includes: Step 5.1: Based on Step 1 and Step 2, extract the global spatial features of the newly added task n. and local spatial features ; Step 5.2, calculate the residuals of task n in the sub-BLS corresponding to the first (n-1) tasks. ,in For the true label of the task, This is the sum of the model predictions for task n in the first (n-1) sub-BLS; Step 5.3: Based on the Fisher information matrix, obtain the output layer weight importance matrix of the first (n-1) sub-BLS. ,in Let be the output layer weight importance matrix of the i-th sub-BLS; combined with the residuals of task n in the sub-BLS corresponding to the first (n-1) tasks in Step 5.

2. And the output layer weight importance matrix, define the sub-BLS objective function corresponding to the nth task as: ,in Spatial mapping features and augmentation features for the nth task This is obtained from Step 1 to Step 3. The output layer weights of the sub-BLS corresponding to the nth task are... For categories of motor imagery tasks, It is a balancing coefficient for the importance of tasks. It is the element-wise Hadamard product. The change in the output layer weights of the sub-BLS corresponding to the nth task is the difference between the output layer weights of the sub-BLS corresponding to the i-th task; this is obtained by diagonalizing the block matrix. diagonal matrix The partial derivatives of the objective function are calculated and set to 0 to obtain the output weights of the nth sub-BLS. expression ,in The output weights for the c-th category are... for The diagonalization matrix; the above steps are to stack the nth BLS-n onto the previous BLS-(n-1) layer in a residual connection manner; Step 5.4, in the feature enhancement layer of sub-BLS-n in Step 5.3, adaptively add enhancement nodes to improve the convergence of the model on the new task. The mapping relationship of the newly added nodes is represented as follows: , and These represent the connection weights and biases of the newly added nodes. For spatial mapping features, For linear mapping, Indicates enhanced features; The spatial mapping features and augmented features after adding augmented nodes are represented as follows: , The spatial mapping features and enhanced features before adding nodes; The newly added enhancement nodes are not shared between subsystems; Step 5.5, repeat Step 4, the encapsulation model is denoted as TiBLS-n, and the output of the encapsulation model is represented as , This represents the output of task n in each sub-BLS layer. Step 5.6: Validate the method using a public dataset.

2. The incremental learning method for motion imagery tasks based on a stacking width learning system according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1 Define the input signal; for the input dataset as follows: , The EEG signal for motor imagery in the i-th task is... Let n be the task category label, where n is the incremental number of tasks. , T is the number of samplings, S is the number of time points in a single sampling, C is the number of lead channels, and L is the number of categories of task i; Step 1.2 Signal preprocessing; Design an 8th-order Butterworth bandpass filter with a bandpass range of 4-40Hz; Perform data augmentation on the original signal using the sliding window method, with a window length of 1s and a sliding step size of 0.2s; Step 1.3 Signal Time-Frequency Decomposition and Sub-Band Signal Reconstruction: The signal filtered in Step 1.2 is subjected to 4-level wavelet packet time-frequency decomposition using wavelet packet transform. The wavelet basis function is Daubechies, the filter length is 4, and the reconstructed signal of 16 sub-bands is obtained by inverse wavelet transform.

3. The incremental learning method for motion imagery tasks based on a stacking width learning system according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1, extract global spatial features; Calculate the wavelet energy of each sub-band. Starting from the band with the highest energy value, search for sub-band energy values ​​alternately in the direction of increasing and decreasing frequency. The optimal sub-band is selected if the sum of the energy values ​​is greater than 0.

9. Using a common-space mode algorithm, the global spatial features of the reconstructed signal in each frequency band are calculated to form the global spatial features. ; Step 2.2: Extract local spatial features; reconstruct all sub-band signals through wavelet inverse transform, calculate the covariance matrix of the reconstructed signal, and map it onto the Riemann manifold, where each covariance matrix is ​​considered a point in the manifold; calculate the Riemann mean as the reference point of the tangent space, project the points in the Riemann manifold onto the tangent space, and take the upper triangular elements to form the tangent space feature vector; adopt a regularization-based supervised feature selection strategy to select the top 10 features most relevant to the task.

4. The incremental learning method for motion imagery tasks based on a stacking width learning system according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1, Multi-view Feature Fusion; For the global and local spatial features from Step 2.1 and Step 2.2, a dual-branch feature mapping layer is simultaneously input, and a sparse autoencoder strategy is used to optimize the mapping features, resulting in the mapping features for Task 1, represented as follows. ,in and These are global and local spatial mapping features, respectively. The number of windows for the feature mapping nodes; Step 3.2, Identify Task 1; The output features of the feature mapping layer The input feature enhancement layer uses the ReLU non-linear activation function to obtain enhanced features. ; Mapping features and enhancement features are represented as ,Establish The connection relationships with the output layer are calculated using ridge regression to determine the connection weights of the output layer. Thus, the input features are obtained. , With model output Based on the mapping relationship, the sub-BLS of recognition task 1 is defined as BLS-1.

5. The incremental learning method for motion imagery tasks based on a stacking width learning system according to claim 4, characterized in that, Step 4 specifically includes: Step 4.1: Calculate the Fisher information matrix of the output weights in BLS-1 to obtain the importance of each output layer connection weight to Task 1. ; Step 4.2, encapsulate the task incremental learning model TiBLS-1 from Task 1, where TiBLS-1 includes the feature construction methods provided in Step 1 and Step 2 and the mapping relationship of BLS-1 in Step 3.

Images