Online brain-computer interface method for enhancing hand fine motion intention recognition based on multi-parameter coupling feature modulation

Abstract

The invention discloses an online brain-computer interface method for enhancing hand fine motion intention recognition performance based on a multi-parameter coupling feature modulation technology. According to the method, a kinematics parameter (slow / fast), a visual auxiliary stimulation parameter (dynamic / static visual auxiliary stimulation) and different action types are introduced to enhance the separability characteristic difference of the hand fine motion intention. An online system extracts sub-band features and constructs feature vectors by using event-related desynchronization features generated by fine motor imagery through a common spatial pattern method of multi-frequency spatial filtering, and then performs fine intention recognition on the extracted features by using a Riemannian mean minimum distance classifier. And an identification result is fed back to the subject in a picture form. According to the method, the problem that the feature separability is not obvious due to the fact that the spatial resolution of the hand fine motor imagery electroencephalogram feature is low is effectively solved, the recognition accuracy of the hand fine motor intention is remarkably improved, and a motor imagery brain-computer interface instruction set is expanded.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interface, and relates to a brain-computer interface method for enhancing the separability of EEG features of hand fine motor intention and improving online recognition performance based on multi-parameter coupling feature modulation technology. Background Art

[0002] Hemiplegia is the most common motor dysfunction in stroke patients, among which fine motor disorders of the hands are particularly common. Although it has been reported that about 82% of stroke patients can recover to the point of independent walking, only 5%-34% of patients can recover full upper limb function. Currently, rehabilitation therapy based on brain-computer interface technology plays a key role in the rehabilitation of stroke patients.

[0003] Motor imagery (MI) is a cognitive process of imagining the movement of a body part without actually moving it. Motor imagery combined with a closed-loop brain-computer interface (BCI) can promote long-term improvement in motor function in chronic stroke patients. Specifically, BCI can serve as a non-muscle communication channel between the user's brain and a computer system for motor rehabilitation. Stroke rehabilitation effectively promotes structural and functional reorganization through myocardial infarction. This is due to the repeated recruitment of motor neuron circuits, repairing the connections between damaged neurons through neuroplasticity, and ultimately improving motor dysfunction.

[0004] Existing MI-BCI technology mainly targets movement patterns that are easy to distinguish in spatial scale, such as the movement imagination of the left hand, right hand, tongue and feet. For the rehabilitation of fine motor skills of the upper limbs, it is usually necessary to identify different movement patterns of different joints of the limb or the same joint, such as hand grasping, elbow extension, multi-finger movement of the hand, etc., which requires accurate recognition of fine motor intentions.

[0005] Imagining the movements of different limbs to obtain peripheral control instructions often leads to inconsistencies between motor imagination tasks and peripheral control instructions, resulting in cognitive disconnection between motor intention and end effector. This cognitive disconnection makes it difficult to provide neural feedback that truly reflects the patient's motor intention, which is not conducive to the induced rehabilitation of motor area nerve cells. Therefore, the development of an online recognition system for fine motor intention that can achieve high-performance EEG encoding and decoding technology is of great significance for the rehabilitation application of fine motor disorders in upper limbs of stroke patients.

[0006] The performance of fine MI-BCI (such as distinguishing different fingers or different gestures) is closer to that of distinguishing different parts such as the hand, wrist, and elbow. The locations of different fingers in the primary motor cortex mapping brain area are closer, and the EEG pattern characteristics generated by their movements are extremely similar. It is necessary to design a reasonable fine MI-BCI experimental paradigm for the hand, combined with an efficient encoding and decoding method, to achieve efficient recognition of fine hand movement intentions.

[0007] As for MI-BCI's decoding of motor intention, it is achieved by detecting the features caused by different MI tasks in the sensorimotor cortex. The accuracy of its system decoding is closely related to the subject's imagination, the rationality of the experimental paradigm design, the feature extraction method, and the effectiveness of the recognition algorithm. In some cases, it is precisely because of the inefficiency of the MI-BCI paradigm design that it is impossible to extract distinguishable features from the subject's brain neural activity; or the feature recognition algorithm cannot fully parse out enough identifiable information, which ultimately makes it difficult for the classification algorithm to identify fine MI movement patterns. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a MI-BCI method based on multi-parameter coupled feature modulation technology to enhance the separability of EEG features of fine hand movements and the accuracy of online recognition of fine intentions, so as to achieve accurate online recognition of fine hand movement intentions.

[0009] The technical solution adopted by the present invention includes:

[0010] The present invention provides an online brain-computer interface method for enhancing the recognition of hand fine motor intention based on multi-parameter coupling feature modulation, which enhances the separability feature difference of hand fine motor intention by introducing kinematic parameters (slow / fast), visual auxiliary stimulation parameters (dynamic / static visual auxiliary stimulation) and different action types. The online system uses the event-related desynchronization features generated by fine motor imagination, extracts sub-band features and constructs feature vectors through the co-spatial pattern method of multi-frequency spatial filtering, and then uses the Riemann mean minimum distance classifier to perform fine intention recognition on the extracted features, and then feeds back the recognition results to the subjects in the form of pictures.

[0011] The method comprises the following steps:

[0012] Experimental paradigm execution: Based on the multi-parameter fine motor intention online identification experimental paradigm, the subjects were guided to complete a series of specified hand fine motor tasks;

[0013] EEG data collection, using an EEG collection system to collect EEG signals obtained by the subjects under a multi-parameter fine motor intention experimental paradigm based on the above design;

[0014] Data preprocessing: filtering the collected EEG signals and downsampling them to improve data quality;

[0015] Feature extraction: extract features from the preprocessed EEG data to extract the features that are most relevant to the fine motor intention;

[0016] Fine motor intention recognition: real-time classification of the subjects’ fine motor imagination EEG data through the model established by the online system;

[0017] Feedback: The fine motor intention recognition results are fed back to the PC and the subjects to help the subjects adjust their status and improve the system recognition effect. It has important practical application value for promoting clinical rehabilitation training for stroke patients.

[0018] The experimental paradigm includes five stages: task preparation, task prompting, motor imagery, feedback, and rest.

[0019] During the task preparation phase, the subject is prompted with “Ready!” for 0-2 seconds and is ready.

[0020] In the task prompting stage, the name of a task is randomly prompted for 2-4 seconds, slow finger pointing or fast grasping, and the subject adjusts the state;

[0021] In the motor imagery stage, the "slow finger pointing" text prompt or "fast grasping" action video that appeared in the task prompt period appears for 4-8 seconds, and the subjects complete the corresponding imagination task according to the prompt;

[0022] In the feedback stage, the system gives feedback on the intention recognition result within 8-10 seconds. When a pointing action is recognized, a pointing picture appears, and when a grasping action is recognized, a grasping picture appears.

[0023] During the rest phase, a prompt “Please take a break!” is given after 10-12 seconds, and the subject takes a break for two seconds.

[0024] Furthermore, the slow finger-contact means that the subject completes a complete action at a frequency of 0.5 Hz, and the fast grasping means that the subject completes a complete action at a frequency of 2 Hz.

[0025] Furthermore, the EEG data acquisition is based on the Neuroscan 64-lead EEG acquisition system to collect EEG signals generated by the subjects under a multi-parameter fine motor intention experimental paradigm.

[0026] Furthermore, the EEG data were collected using a Neuroscan SynAmps2 amplifier with 60 electrodes, including FP1, FPZ, Fp2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCZ, FC2, FC4, FC6, FT8, T7, C5, C3, C1, CZ, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPZ, CP2, CP4, C P6, TP8, P7, P5, P3, P1, PZ, P2, P4, P6, P8, PO7, PO5, PO3, POZ, PO4, PO6, PO8, O1, OZ, O2 collected the EEG activity of the head, where the ground electrode was located in the middle between the Fpz and Fz electrodes, and the reference electrode was placed between the frontal lobe and the parietal lobe. The impedance of all electrodes was controlled below 5 kilo-ohms, and the layout of the electrodes followed the internationally accepted 10-20 system standard. The bandpass filter range was set to 1-200 Hz, and a 50 Hz notch filter was applied.

[0027] Furthermore, the feature extraction uses a filter bank co-spatial pattern algorithm to extract two features that are most correlated with the intent.

[0028] Furthermore, the fine motor intention recognition uses a model established by a Riemann mean minimum distance classifier to perform real-time classification on the features extracted from the subject's fine motor imagery EEG data.

[0029] Furthermore, the multi-parameter fine motor intention online recognition experimental paradigm design includes multi-parameter selection, fine motor intention design and experimental process design.

[0030] Among them, multi-parameter selection is used to select a variety of different feature modulation parameters, mainly including movement speed, type, and visual auxiliary stimulation parameters; fine motor intention design is used to design different types of fine motor intentions of the hand, such as the thumb and index finger pointing movement and grasping movement of the right hand, to simulate actual movement intentions; experimental process design is used to combine the selected multi-parameters and fine motor intentions into a standardized experimental paradigm to ensure the consistency and repeatability of the experiment.

[0031] Advantages and beneficial effects of the present invention:

[0032] 1. This invention introduces feature modulation enhancement technology into the online recognition of hand fine motor intention, and combines multi-parameter coupling modulation fine motor intention features to achieve online accurate recognition of hand fine motor intention. It is of great significance for the rehabilitation application of upper limb fine motor disorders in stroke patients.

[0033] 2. The present invention effectively improves the problem of unclear feature separability due to the low spatial resolution of the EEG features of fine motor imagery of the hand, and significantly improves the accuracy of fine motor intention recognition of the hand, and expands the motor imagery brain-computer interface instruction set. This method provides new ideas and technical means for improving the performance of the fine motor imagery brain-computer interface online system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of an online brain-computer interface system for enhancing hand fine motor intention recognition based on multi-parameter coupled feature modulation according to an exemplary embodiment of the present disclosure is shown;

[0035] FIG2( a ) shows a single trial experiment flow chart of a multi-parameter fine motor intention online experimental paradigm according to an exemplary embodiment of the present disclosure;

[0036] FIG2( b ) shows a schematic diagram of frequency representing speed according to an exemplary embodiment of the present disclosure, where the left figure represents a slow speed and the right figure represents a fast speed;

[0037] Figure 3 A feature extraction flow chart according to an exemplary embodiment of the present disclosure is shown;

[0038] Figure 4 A flow chart of online system data processing according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] The following is a detailed description of an online brain-computer interface method for enhancing the recognition of fine hand movement intention based on multi-parameter coupling feature modulation in conjunction with the embodiments and drawings.

[0040] Example 1

[0041] like Figure 1 As shown, the structure diagram of the online brain-computer interface system for enhancing the recognition of hand fine motor intention based on multi-parameter coupled feature modulation of the present invention includes the design of a multi-parameter fine motor intention online recognition experimental paradigm, EEG data acquisition, preprocessing, feature extraction, fine motor intention recognition and feedback;

[0042] The experimental paradigm design of multi-parameter fine motor intention online recognition includes: experimental parameter selection, fine motor intention design and experimental process design.

[0043] Multi-parameter selection is used to select a variety of different parameters, including speed, type and visual auxiliary stimulation;

[0044] Fine motor intention design: two motor intentions were designed: slow finger pointing and fast grasping of the right hand. The actual actions corresponding to the two intentions were presented in static text and dynamic video respectively. The experimental design combined the speed, type and visual auxiliary stimulation parameters.

[0045] The experimental process design is used to combine the selected multiple parameters and fine movements into a standardized experimental paradigm to ensure the consistency and repeatability of the experiment;

[0046] The EEG data acquisition is used to acquire EEG signals obtained by the experimenter under the multi-parameter fine motor intention experimental paradigm based on the design based on the EEG acquisition system;

[0047] The preprocessing is used to filter and downsample the collected EEG signals to improve data quality;

[0048] Feature extraction is used to extract features from the preprocessed EEG data using the filter bank common spatial pattern (FBCSP) algorithm to extract the two features that are most relevant to the intention;

[0049] Fine motor intention recognition: A model built using the minimum distance to riemannianmean (MDRM) classifier is used to perform real-time classification of features extracted from the subjects’ fine motor imagery EEG data.

[0050] Feedback is used to feed back the fine motor intention recognition results to the PC and subjects in the form of action pictures, helping the subjects adjust their status and improve the system recognition effect.

[0051] As shown in Figure 2(a), the multi-parameter fine motor intention experimental paradigm includes five stages: task preparation, task prompting, motor imagery, feedback, and rest.

[0052] During the task preparation phase, the subject was prompted with “Ready!” for 0-2 seconds and was ready;

[0053] In the task prompting stage, the name of a task (slow finger-pointing or fast grasping) was randomly prompted for 2-4 seconds, and the subjects adjusted their state;

[0054] During the motor imagery phase, the “slow pointing” text prompt or “fast grasping” action video that appeared during the task prompt period appeared for 4-8 seconds. During this phase, the subjects completed the corresponding imagination task according to the prompt.

[0055] In the feedback stage, the system will give feedback on the intention recognition results within 8-10 seconds. When a pointing action is recognized, a pointing picture will appear; when a grasping action is recognized, a grasping picture will appear.

[0056] During the rest phase, a prompt “Please take a break!” is displayed after 10-12 seconds, and the subjects take a break for two seconds. Considering the amount of data collected and the total duration of the experiment, this phase can be set to a random value between 2 and 4 seconds.

[0057] This is a one-trial experimental process, and three parameters, namely speed, type and visual auxiliary stimulation, were introduced in the motor imagery stage, forming an experimental paradigm for online identification of multi-parameter fine motor intentions.

[0058] As shown in Figure 2(b), the slow finger-pointing subjects completed a complete movement at a frequency of 0.5 Hz, that is, one movement in two seconds, and the fast grasping subjects completed a complete movement at a frequency of 2 Hz, that is, one movement in 0.5 seconds.

[0059] like Figure 3 The figure shows the process of motion intention feature extraction based on FBCSP. The basic principle is to divide the frequency band of 8-26Hz into 5 sub-bands (i.e. 8-14Hz, 11-17Hz, 14-20Hz, 17-23Hz, 20-26Hz) by designing a third-order Butterworth bandpass filter. These sub-bands mainly include alpha and beta bands, which are 8-13 and 13-26Hz respectively. Then, the EEG components of each frequency band are spatially filtered and feature extracted. The mutual information between features is calculated using the feature vector corresponding to the filtered signal, and they are arranged in order from large to small. The two features with the largest correlation are selected according to the mutual information as the feature data of FBCSP for the construction of the subsequent classifier.

[0060] Fine-grained intent recognition uses the MDRM classifier to classify the extracted features, which mainly involves the following steps:

[0061] (1) Calculate the covariance matrix

[0062] For a given feature matrix X (the feature matrix size of each trial is n×m, where n is the number of samples and m is the number of channels), the calculation formula of the covariance matrix C is:

[0063] where μ is the mean of the feature matrix X.

[0064] (2) Calculate the Riemann mean

[0065] The calculation of the Riemann mean usually uses an iterative method, which can be approximated by the following steps: The initial Riemann mean C mean Can be initialized to any covariance matrix: C mean =C 1 The Riemann mean C is calculated using the following update rule mean Until convergence:

[0066]

[0067] (3) For two given covariance matrices C i and Cj , the Riemann distance between them is calculated as:

[0068] where λ k (C i ,C j ) is the matrix C i and C j The generalized eigenvalues ​​of .

[0069] (3) Classification

[0070] To complete the classification, the covariance matrix C of each test sample is testi , calculate its distance from each type of Riemann mean, and select the category with the minimum distance:

[0071]

[0072] Among them C mean,c is the Riemann mean of class c.

[0073] like Figure 4 The figure shows the specific process of data processing. The experimental data is divided into a training set and a test set. The former is used to train the classifier model. The data volume is 60×4000×40 (60 is the number of leads, 4000 is the number of sampling points, and 40 is the total number of experiments). After the original data is transferred from the Scan software to the PC, dimensionality reduction is performed in MATLAB. The filter is applied to the preprocessed data for spatial filtering, and the filter is saved in the specified path. After the spatial filter is constructed, the signal features are extracted, and the two feature data with the highest relevance to the task are input into the MDRM model to complete the classifier modeling. The classifier and spatial filter are saved in the same folder for subsequent testing.

[0074] The testing phase is the verification of the classifier model, which is the core of the real-time brain-computer interface system and the main difference from the offline system. At this stage, the system analyzes and classifies single EEG signals in real time and feeds back the prediction results. After the test set data is preprocessed, it is filtered through the designed spatial filter, and the feature data is then input into the established classifier model for classification prediction. The results are displayed in real time on the stimulation prompt interface of the client PC. The quality of the real-time classification results directly affects the performance of the brain-computer interface system, which is closely related to the quality of the classifier model. Therefore, in the real-time experiment, the subjects are required to concentrate their attention during the training phase and perform the imagination task carefully in order to build a high-precision classifier model.

[0075] The feedback link presents the classification results to the subjects in a visual manner. The classification results obtained by the MDRM model are displayed in the form of numbers 1 and 2 (1 represents finger pointing, 2 represents hand grasping). However, feedback in digital form cannot fully reflect the effect, so this study uses pictures to provide result feedback. When the classification result is "1", a finger pointing picture is displayed; when it is "2", a grasping picture is displayed. After the picture is displayed for a period of time, the subject rests for a few seconds, the feedback ends, and the imagination task of the next trial begins. The test results of the online system show that the average classification accuracy of online identification of two types of hand fine motor intentions, finger pointing and grasping, reached 74.00% ± 3.57%, exceeding the international standard of 70.00%. The feasibility of the online recognition method of fine motor intentions with multi-parameter coupling modulation proposed in the present invention is proved.

[0076] As shown in Table 1, this is the average result obtained by testing 15 subjects using the MDRM model in the early stage of the present invention.

[0077] The experiment designed eight hand fine MI experimental tasks, namely: static text prompt-slow-finger pointing, static text prompt-fast-finger pointing, static text prompt-slow-hand grasping, static text prompt-fast-hand grasping, dynamic visual auxiliary stimulation-slow-finger pointing, dynamic visual auxiliary stimulation-fast-finger pointing, dynamic visual auxiliary stimulation-slow-hand grasping, dynamic visual auxiliary stimulation-fast-hand grasping.

[0078] In the experiment, static text prompts and slow speed are called basic experimental conditions, and dynamic visual auxiliary stimulation and fast speed are called parameter modulation experimental conditions. The experimental task with only static text prompts and slow speed parameters is called a single parameter (type) experimental task, the experimental task with dynamic visual auxiliary stimulation or fast speed parameters (and types) is called a two-parameter experimental task, and the experimental task with dynamic visual auxiliary stimulation and fast speed parameters (and types) is called a three-parameter experimental task.

[0079] The experimental task volume is to complete 40 single-trial experiments for each experimental task.

[0080] Table 1 shows the statistical results of the highest average accuracy in single-parameter experimental tasks, dual-parameter experimental tasks, and three-parameter experimental tasks.

[0081] Table 1

[0082]

[0083] It can be seen from Table 1 that the average classification accuracy under the three-parameter experimental task (static text prompt-slow-finger vs. dynamic visual auxiliary stimulation-fast-hand grasping) is the highest, which is 82.19%±3.67% under the MDRM model. Compared with the fine motor intention recognition accuracy under the single-parameter experimental task (static text prompt-slow-finger vs. static text prompt-slow-hand grasping) and the dual-parameter experimental task (static text prompt-fast-finger vs. static text prompt-slow-hand grasping and static text prompt-slow-finger vs. dynamic visual auxiliary stimulation-slow-hand grasping), it is improved by 12.76%, 8.22% and 2.04% respectively. The results prove that the multi-parameter coupling modulation method proposed in the present invention combined with the MDRM model can effectively enhance the intention recognition accuracy of different fine movements of the hand.

[0084] In summary, the embodiment of the present invention designs a brain-computer interface method for enhancing online recognition of hand fine motor intention based on multi-parameter coupling feature modulation, aiming to accurately and quickly identify hand fine motor intention. The method realizes the recognition of fine motor imagination tasks by introducing kinematic parameters (speed) and visual auxiliary stimulation parameters (dynamic visual auxiliary stimulation) and action type multi-parameter feature modulation technology. Utilizing the event-related desynchronization features generated by fine MI, the sub-band features are extracted and the feature vector is constructed by the co-spatial pattern method of the multi-frequency spatial filter group, and then the extracted multi-parameter features are classified by the MDRM classifier to improve the online recognition accuracy of the hand fine motor intention. The present invention effectively improves the problem of unclear feature separability caused by the low spatial resolution of hand movements, solves the limitation of a small set of single limb movement instructions, and significantly improves the accuracy of hand fine motor intention recognition. The online system verifies the reliability of the method. The method provides new ideas and technical means for improving the performance of fine MI-BCI online systems.

[0085] Unless otherwise specified, the types of devices involved in the embodiments of the present invention are not limited as long as they can achieve the functions described.

[0086] The present invention describes only an embodiment attempted during the research process of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent substitution or improvement made within the spirit and basic principles of the present invention should be considered to be within the protection scope of the present invention.

Claims

1. An online brain-computer interface method for enhancing hand fine motor intention recognition based on multi-parameter coupling feature modulation, characterized in that: The method comprises the following steps: Experimental paradigm execution: Based on the multi-parameter fine motor intention online identification experimental paradigm, the subjects were guided to complete a series of specified hand fine motor tasks; EEG data collection, using an EEG collection system to collect EEG signals obtained by the subjects under a multi-parameter fine motor intention experimental paradigm based on the above design; Data preprocessing: filtering the collected EEG signals and downsampling them to improve data quality; Feature extraction: extract features from the preprocessed EEG data to extract the features that are most relevant to the fine motor intention; Fine motor intention recognition: real-time classification of the subjects’ fine motor imagination EEG data through the model established by the online system; Feedback: Feedback the fine motor intention recognition results to the PC and the subjects to help the subjects adjust their status and improve the system recognition effect; The experimental paradigm includes five stages: task preparation, task prompting, motor imagery, feedback, and rest. During the task preparation phase, the subject is prompted with "Ready!" for 0-2 seconds to get ready. In the task prompting stage, the name of a task is randomly prompted for 2-4 seconds, slow finger pointing or fast grasping, and the subject adjusts the state; In the motor imagery stage, the "slow finger pointing" text prompt or "fast grasping" action video that appeared in the task prompt period appears for 4-8 seconds, and the subjects complete the corresponding imagination task according to the prompt; In the feedback stage, the system gives feedback on the intention recognition result within 8-10 seconds. When a pointing action is recognized, a pointing picture appears, and when a grasping action is recognized, a grasping picture appears. During the rest phase, the prompt "Please take a break!" is given after 10-12 seconds, and the subject takes a break for two seconds.

2. The method according to claim 1, characterized in that The slow finger-contact refers to a complete movement of the subject at a frequency of 0.5 Hz, and the fast grasp refers to a complete movement of the subject at a frequency of 2 Hz.

3. The method according to claim 1, characterized in that The EEG data acquisition is based on the Neuroscan 64-lead EEG acquisition system to collect EEG signals generated by the subjects under a multi-parameter fine motor intention experimental paradigm.

4. The method according to claim 3, characterized in that The EEG data were collected using a Neuroscan SynAmps2 amplifier with 60 electrodes to collect EEG activity from the head.

5. The method according to claim 1, characterized in that: The feature extraction uses a filter bank co-spatial pattern algorithm to extract the two features that are most correlated with the intention.

6. The method according to claim 1, characterized in that The fine motor intention recognition uses a model established by a Riemann mean minimum distance classifier to perform real-time classification of features extracted from the subject's fine motor imagery EEG data.

Images