Multi-person movement imagery recognition method and brain-computer system based on cross-brain fusion decision

Abstract

The present invention discloses a multi-person motion imagination recognition method and a brain-computer system based on cross-brain fusion decision. The present invention is based on a multi-person motion imagination recognition method based on cross-brain fusion decision, which is specifically as follows: 1. n trainees perform motion imagination; 2. The EEG data obtained in step 1 are pre-processed and labeled respectively. 3. The EEG data of each trainee in the training set are fused to obtain the training fused EEG data corresponding to each motion imagination task. 4. The classifier is trained with the labeled training fused EEG data. 5. n test subjects imagine the same motion, and the EEG signals of each test subject are collected while imagining, and pre-processed. Afterwards, the EEG data of the n test subjects are fused to measure the cross-brain fused EEG data; the cross-brain fused EEG data is imported into the trained classifier for classification and judgment, and the motion categories imagined by the n test subjects are obtained.

Description

Technical Field

[0001] The patent of this invention belongs to the field of artificial intelligence-pattern recognition-brain-computer interface, and specifically relates to an online brain-computer interface system for multi-person motion imagination based on cross-brain fusion decision-making. Background Art

[0002] Brain Computer Interface (BCI) is considered as one of the important contents of brain plans of various countries. BCI refers to a technology that can establish a new information exchange method between the human brain and the external environment that does not rely on peripheral nerves or muscle tissue. Its working principle is to collect human brain signals, perform pattern recognition on them, analyze the brain's intentions, and then convert them into external commands, so as to achieve communication and interaction with external devices / environments. Brain-computer interface based on electroencephalogram (EEG) signals has important advantages such as low cost, easy operation and non-invasiveness, and is the main direction of BCI at present.

[0003] The three major paradigm technologies of BCI include P300, motor imagery, and steady-state visual evoked. Among them, motor imagery refers to the use of the brain's thoughts to imagine limb movements directly without actual physical behavior, and then transform them into subsequent actual operations. It is an endogenous spontaneous EEG. Unlike induced EEG, it does not require external stimulation. As long as people imagine movements, the brain waves will show specific waveforms. Therefore, motor imagery BCI is widely used: for limb inconveniences caused by paralysis, stroke and other diseases common in the elderly, it can not only help patients control objects and achieve self-care; it can also be used as a means of rehabilitation therapy to help patients restore their motor abilities to the maximum extent. However, the current brain-computer interface system has the following three defects:

[0004] (1) Low recognition accuracy: Since the functions and mechanisms of tens of billions of neurons in the brain have not been sufficiently understood and explored, BCI technology is limited in its ability to correctly recognize brain signals. Moreover, brain signals are subject to multiple interferences from the environment and the brain's own physiological signals, making it difficult to recognize task-related signals.

[0005] (2) Weak stability: The current motor imagery brain-computer interface mainly relies on single-player mode, and the system’s decisions come only from a single individual, which is prone to misjudgment. At the same time, as time goes by, the user’s mental state is also prone to change, such as fatigue, lack of concentration, and other problems.

[0006] (3) Low online level: Existing motor imagery brain-computer interfaces usually only perform offline data analysis, which separates data collection and algorithms, resulting in BCI research remaining in the experimental stage. However, actual applications require that BCI must be an online system to improve the real-time decoding of brain intentions. Summary of the invention

[0007] The purpose of this patent is to overcome the shortcomings of the existing technology and provide an online brain-computer interface system for multi-person motion imagination based on cross-brain fusion decision-making. The system supports simultaneous use by multiple people and improves the stability and recognition accuracy of online decision-making.

[0008] The present invention is based on a multi-person motion imagery recognition method based on cross-brain fusion decision-making, and the specific steps are as follows:

[0009] Step 1: Provide multiple motor imagery tasks to n trainees, where n≥2; the multiple motor imagery tasks include all m task categories, where m≥2. Collect EEG signals while each trainee performs motor imagery, and obtain EEG signals of each trainee when performing each motor imagery task.

[0010] Step 2: Preprocess the EEG data obtained in step 1 and label the EEG signals with corresponding motor imagery task labels to form a training set.

[0011] Step 3: Fuse the EEG data of each trainee in the training set to obtain the fused EEG data for training corresponding to each motor imagery task.

[0012] Step 4: Train the classifier using the labeled training fused EEG data.

[0013] Step 5: n subjects imagine the same movement, and collect the EEG signals of each subject while imagining, and perform preprocessing. After that, the EEG data of the n subjects are fused to measure cross-brain fused EEG data; the cross-brain fused EEG data is imported into the trained classifier for classification and judgment, and the movement categories imagined by the n subjects are obtained.

[0014] Preferably, in step three, the fusion of EEG data adopts the following method: averaging the EEG data of n subjects.

[0015] Preferably, in step three, the fusion of EEG data adopts the following method: splicing the EEG data of n subjects.

[0016] Preferably, the classifier described in step 4 includes m binary classifiers arranged side by side. Among the two categories of the m binary classifiers, one category is a single motor imagination category, and the other category is a fusion category of all motor imagination categories outside the single motor imagination category. The single motor imagination category of each binary classifier is different. After the EEG data is input into the m binary classifiers; the m binary classifiers respectively output a feature matrix after classification. The m feature matrices are spliced ​​and imported into a single classifier to output the classification result.

[0017] Preferably, the specific process of training the classifier is as follows: use the training set to train m binary classifiers separately; during training, according to the preset classification requirements of the binary classifier, first merge the m-1 motor imagination categories into one category, and use the last motor imagination category as another category for binary classification.

[0018] The training process of a single binary classifier is as follows: Calculate the sample covariance N1 and N2 of the two categories of the binary classifier. The calculation formula is as follows:

[0019]

[0020]

[0021] Among them, E1 and E2 are the EEG data corresponding to the two categories of the binary classifier in the training set; trace(·) is the trace operation of the matrix.

[0022] After principal component decomposition, whitening, spatial filtering, and projection calculation of sample covariance N1 and N2, the projection matrix W of the binary classifier is obtained. M×M ; Calculate the classification feature matrix Z M×N =W M×M E M×N Among them, E M×N is all the data in the training set.

[0023] Finally, the feature matrices corresponding to the m binary classifiers are concatenated and imported into the classifier.

[0024] Preferably, in step one, the task instruction time for each motor imagery task is 0.5 s, the motor imagery time is 4 s, and the time interval between two motor imagery tasks is 1 s.

[0025] Preferably, in step 1, a 64-electrode Neuroscan device is used to acquire EEG data with a sampling frequency of 1000 Hz.

[0026] Preferably, in step one, the motor imagery task is presented to the subject via video or picture.

[0027] Preferably, the preprocessing in step 2 includes bandpass filtering and removing the average reference.

[0028] An online brain-computer interface system for realizing a motor imagery training method for cross-brain fusion decision-making includes a central processing unit PC1, a central processing unit PC2, a central processing unit PC3 and a central processing unit PC4. The central processing unit PC1 and the central processing unit PC2 are both provided with a multi-person EEG synchronous acquisition module and a preprocessing module, and the multi-person EEG synchronous acquisition module acquires the user's EEG signal and transmits it to the preprocessing module for preprocessing. The central processing unit PC4 is provided with a task module and a label recognition module; the task module provides a motor imagery task to the trainee during training. The label recognition module generates a label.

[0029] The central processing unit PC3 is provided with a data transmission and communication module, a feature calculation module, a multi-person fusion analysis and decision module, and a visualization module. The EEG signal output by the preprocessing module is labeled by the label recognition module and then output to the feature calculation module through the data transmission and communication module. The feature calculation module calculates the classification feature matrix of the labeled EEG data. The classifier in the multi-person EEG signal synchronization acquisition module is trained using the classification feature matrix generated by the feature calculation module, and after training, the motor imagery category of the multi-person fused EEG signal is identified, and the identification result is displayed on the visualization module.

[0030] Compared with the prior art, the present invention has the following advantages and effects:

[0031] 1. The motor imagery method for cross-brain fusion decision-making provided by the present invention collects EEG data of multiple users through EEG equipment, fuses the data of multiple users, and calculates the decision accuracy online, which can significantly improve the accuracy of motor imagery recognition and is conducive to the development of new multi-person hybrid brain-computer interfaces.

[0032] 2. The present invention has the functions of synchronously recording signals from multiple multi-channel EEG acquisition devices, analyzing and identifying cross-brain signals, and visualizing decision results. The differences from the traditional motor imagery brain-computer interface are as follows: the present invention realizes cross-brain decision fusion, making decisions more accurate; it is a real-time online system with good experimental results, the decision method can be changed arbitrarily, the selection is more flexible, and the analysis process is simplified; the operation process is simple, multiple PCs communicate through a local area network, and the visualization of decision results and the adjustment of decision methods can be realized through the system's user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a system framework diagram of the present invention;

[0034] Figure 2 is a system flow chart of the present invention;

[0035] Figure 3 This is a schematic diagram of the multi-person brain-computer interface system of the patent of this invention. DETAILED DESCRIPTION

[0036] The invention content of the present invention will be explained in detail below with reference to the accompanying drawings and an example in which two specific users participate, but the implementation method and number of users of the present invention are not limited thereto.

[0037] like Figure 1 , 2 As shown in Figure 3, an online brain-computer interface system for implementing a motor imagery training method for cross-brain fusion decision-making includes central processing units PC1, PC2, PC3 and PC4. The central processing units PC1 and PC2 are both provided with a multi-person EEG synchronous acquisition module and a preprocessing module, and the multi-person EEG synchronous acquisition module acquires the user's EEG signal and transmits it to the preprocessing module for preprocessing.

[0038] The central processing unit PC4 is provided with a task module and a label recognition module; the task module displays the motor imagery task to the user during training. The label recognition module generates a label (ie, task type) according to the task displayed by the task module.

[0039] The central processing unit PC3 is provided with a data transmission and communication module, a feature calculation module, a multi-person fusion analysis and decision module, and a visualization module. The EEG signal output by the preprocessing module is labeled by the label recognition module and then output to the feature calculation module through the data transmission and communication module. The feature calculation module calculates the feature matrix of the labeled EEG data. The classifier in the multi-person EEG signal synchronization acquisition module is trained using the feature matrix generated by the feature calculation module, and after training, the motor imagery category of the multi-person fused EEG signal is identified, and the identification result is displayed on the visualization module.

[0040] The brain-computer interface system performs a multi-person motion imagery recognition method based on cross-brain fusion decision-making. The specific process is as follows:

[0041] Step S1: Task module:

[0042] Set and generate a time interval sequence and a task sequence, and maintain a timer; in this embodiment, the time interval in the time interval sequence is 1s, the task indication time is 0.5s, and the motor imagery time is 4s. In the motor imagery task design, in order to ensure the training effect of the brain, the training task is designed as a block task including alternating rest phases and task phases. The timer counts down according to the time in the time interval sequence. When the timer is finished, the current indication screen is modified according to the task sequence to indicate that the user enters the rest phase or the task phase; at the same time, the start time point and end time point of the rest phase and the task phase are notified to the data preprocessing module. In this embodiment, three motor imagery tasks are set, namely "left hand grasping", "right hand grasping" and "idle state".

[0043] Step S2: Multi-person EEG synchronous acquisition module: synchronize the EEG signals acquired by two multi-channel EEG devices. In this embodiment, a 64-electrode Neuroscan device is used to acquire EEG data. The sampling frequency is 1000 Hz. The electrode cap adopts the international 10 / 20 system electrode placement method. The 64 electrodes are FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCZ, FC2, FC4, FC6, FT8, T7, C5, C3, C1, CZ, C2, C4, C6, T8, M1, TP7, CP5, CP3, CP1, CPZ, CP2, CP4, CP6, TP8, M2, P7, P5, P3, P1, PZ, P2, P4, P6, P8, PO7, PO5, PO3, POZ, PO4, PO6, PO8, CB1, O1, OZ, O2, and CB2. Two reference electrodes (i.e., M1 and M2) are placed on the left and right earlobes respectively. Another EKG electrode is placed on the left hand artery to record ECG data. The locations of the 62 EEG channels in addition to the ECG and reference electrodes. Figure 3 As shown. Let two users (respectively denoted as user A and user B) sit side by side, user A's EEG acquisition device is connected to PC1; user B's EEG acquisition device is connected to PC2. The two users sit side by side and watch the motor imagery prompt screen on PC4 at the same time.

[0044] Step S3: Preprocessing module: receiving EEG data from the EEG acquisition module.

[0045] The collected EEG data is preprocessed to reduce the interference of artifacts. The specific process includes: (1) Bandpass filtering: Create a bandpass filter to extract the 2-47 Hz EEG component from the selected EEG data. This embodiment uses a 2nd-order Butterworth filter to implement filtering. (2) De-averaging reference: Calculate the average value of the EEG data of 15 channels except the ECG after filtering, and subtract the average value from the data of each EEG channel.

[0046] Step S4: Label recognition module: Real-time detection of the imagination task prompts of the computer running Eprime, and based on different prompt tasks, the corresponding labels are simultaneously added to the EEG data of each user.

[0047] Step S5: Data transmission module: integrating data and transmitting data, specifically including:

[0048] 1) Create an "IPC" (named pipe) on PC3 to provide shared memory for PC1 and PC2 to achieve communication between PC1 and PC2 across the network.

[0049] 3) PC1, PC2, PC3 and PC4 establish a connection based on the SMB (Server Message Block) protocol.

[0050] 2) PC1 and PC2 transmit the EEG data read online to PC3 in slices of a set length. In this embodiment, we use a 0.5 second time window, that is, 0.5 seconds of EEG data are sent every second, and PC4 transmits the label to PC3. PC3 labels the received EEG data of user 1 and user 2 to form a training set.

[0051] 3) Read data and tags on PC3.

[0052] Step S6: Energy characteristic calculation module, specifically including:

[0053] 1) Establish a three-classifier model; the three-classifier model includes three binary classifiers set side by side. Each of the three binary classifiers has one classification category that is a single motor imagery category, and the other category is a fusion category other than the single motor imagery category corresponding to itself. After the EEG data is input into the three binary classifiers; the three binary classifiers each output a feature matrix. After splicing the three feature matrices, import them into the SVM to output the classification result.

[0054] 2) Multi-category common spatial potential (CSP) was used to calculate the most distinguishable energy distribution feature vectors for the three types of motor imagery tasks, as follows:

[0055] The three binary classifiers were trained separately using the training set. During the training, according to the preset classification requirements of the binary classifier, the two motor imagery categories were first merged into one of the categories, and the third motor imagery category was used as the other category for binary classification CSP. In this way, a total of three binary classification CSPs were performed to achieve the expansion of binary classification to three classifications.

[0056] Calculate the sample covariance N1 and N2 of the two categories of the binary classifier. The calculation formula is as follows:

[0057]

[0058]

[0059] Among them, E1 and E2 are the EEG data corresponding to the two categories of the binary classifier in the training set; trace(·) is the trace operation of the matrix.

[0060] After principal component decomposition, whitening, spatial filtering, and projection calculation of sample covariance N1 and N2, the projection matrix W of the binary classifier is obtained. M×M

[0061] Further calculation of classification feature matrix such as Z M×N as follows:

[0062] Z M×N =W M×M E M×N

[0063] Among them, E M×N is all the data in the training set.

[0064] In this example, we use a 1 second time window to calculate the CSP of the selected channel.

[0065] 2) Using the EEG signals of the two users as training sets, the three classifiers are trained according to the above method to obtain a single-brain three-classifier.

[0066] 3) Calculate the cross-brain data layer fusion-CSP features, which are processed by one of the following two methods:

[0067] ①. By averaging the EEG data of two people, we get a cross-user fusion data layer, and then train the three classifiers according to the above method to get a cross-brain three-classifier.

[0068] ②. Splice the EEG data of two people to increase the number of channels and data volume, improve the CSP projection performance, and then train the three classifiers according to the above method to obtain a cross-brain three-classifier.

[0069] 4) Use the same method to calculate the eigenvectors of all experimental trials.

[0070] Step S7: Use a single user to perform motor imagery to test the classification accuracy of the single-brain three-classifier. Use two users to perform motor imagery and fuse the EEG signals to test the classification accuracy of the cross-brain three-classifier.

[0071] Step S8: The two subjects imagine the same movement; the EEG signals of the two subjects are collected according to the method in step S2. The collected EEG data of the two subjects are averaged or spliced ​​to obtain a fused EEG data layer; the obtained fused EEG data layer is input into the trained three-classifier model to determine the movement type imagined by the two subjects.

[0072] The present invention can significantly improve the accuracy of EEG acquisition of motor imagery instructions, so that it can be used in scenarios such as EEG control, and can also be applied to the mining of social cognition multi-person interaction mechanisms. Specific application scenarios are as follows:

[0073] 1. Complex mission scenarios: Multiple people collaborate to complete the same control goal. For example, this system can be embedded in a drone system to improve the control of the drone and display the accuracy of the brain's control intention in real time.

[0074] 2. Physical exercise training: A new mode of exercise training with multi-person interaction. When multiple people participate in a task, such as completing a goal or multiple goals together, there is a natural interaction between them, achieving a compound feedback adjustment between man, machine and man, thereby improving the efficiency of the user's physical exercise.

[0075] 3. Robustness - Asynchronous brain-computer interface: Through the characteristic indicators of multi-person motion imagination, the user's task participation status that is difficult to determine by a single person can be realized. For example, it is difficult to determine the idle state of a user with active brains, and the superposition of multiple people's EEG makes it easier to distinguish the idle state, thereby better realizing asynchrony.

[0076] 4. Games: multiplayer battles, man-machine battles, etc. This system has been applied to multiplayer football games. Multiple people cooperate to fight against the machine, and control the movement of the ball by imagination, and control the ball into the goal to win the game.

[0077] 5. Explore the brain mechanisms of psychiatric diseases such as social disorders. Borderline personality disorder is the most common social cognitive disorder, characterized by repeated interpersonal conflicts and unstable social relationships, but the neurobiological mechanism of social disorders in patients remains unclear. This system was used to perform social interaction tasks on patients and healthy controls, and the corresponding EEG signals were recorded at the same time to explore the neural mechanisms of social disorders in patients.

[0078] In order to verify the advantages of the present invention over the prior art, we identified the classification accuracy based on single-person and double-person cross-brain motor imagery data (10 groups, a total of 20 people). Among them, each single user (the single person in the same group is represented as user 1 and user 2) uses 64-channel EEG data, and the two methods mentioned above are used to extract CSP features for the double-person cross-brain data, and finally the features are classified using SVM. The results are shown in Table 1. From the results, it can be seen that under different fusion methods, the classification accuracy based on multiple brains is significantly higher than the classification accuracy based on single brain.

[0079] Table 1 Classification results (accuracy) of single user and different multi-person cross-brain fusion methods

[0080]

[0081]

Claims

1. The application of multi-person motion imagery recognition method based on cross-brain fusion decision in UAV control is characterized by: The process of the multi-person motion imagery recognition method is as follows: Step 1: providing multiple motor imagery tasks to n trainees, where n≥2; the multiple motor imagery tasks include all m task categories, where m≥2; collecting EEG signals while each trainee performs motor imagery, and obtaining EEG signals of each trainee when performing each motor imagery task; Step 2: Preprocess the EEG data obtained in step 1 respectively; and label the EEG signals with corresponding motor imagery task labels to form a training set; Step 3: Fusing the EEG data of each trainee in the training set to obtain fused EEG data for training corresponding to each motor imagery task; the EEG data fusion is performed by splicing or averaging the EEG data of n testees; Step 4, training the classifier with labeled training fused EEG data; the classifier includes m binary classifiers arranged side by side; among the two categories of the m binary classifiers, one category is a single motor imagery category, and the other category is a fusion category of all motor imagery categories other than the single motor imagery category; the single motor imagery category of each binary classifier is different; after the EEG data is input into the m binary classifiers; after the m binary classifiers are classified, each outputs a feature matrix; the m feature matrices are spliced ​​and imported into a single classifier to output the classification result; The specific process of training the classifier is as follows: m binary classifiers are trained separately using the training set; during training, according to the preset classification requirements of the binary classifier, m-1 motor imagery categories are first merged into one category, and the last motor imagery category is used as another category for binary classification; The training process of a single binary classifier is as follows: Calculate the sample covariance N1 and N2 of the two categories of the binary classifier. The calculation formula is as follows: Among them, E1 and E2 are the EEG data corresponding to the two categories of the binary classifier in the training set; trace(·) is the trace operation of the matrix; After principal component decomposition, whitening, spatial filtering, and projection calculation of sample covariance N1 and N2, the projection matrix W of the binary classifier is obtained. M×M ; Calculate the classification feature matrix Z M×N =W M×M E M×N ; Among them, E M×N is all the data in the training set; Finally, the feature matrices corresponding to the m binary classifiers are concatenated and imported into the classifier; Step 5: n subjects imagine the same movement, and the EEG signals of each subject are collected and preprocessed while imagining; then, the EEG data of the n subjects are fused to measure cross-brain fused EEG data; the cross-brain fused EEG data are imported into the trained classifier for classification and judgment, and the movement categories imagined by the n subjects are obtained.

2. The application of the multi-person motion imagination recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: In step three, the EEG data are fused in the following way: the EEG data of n subjects are averaged.

3. The application of the multi-person motion imagination recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: In step three, the EEG data are fused in the following way: the EEG data of n subjects are spliced.

4. The application of the multi-person motion imagination recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: In step 1, the task instruction time for each motor imagery task was 0.5 s, the motor imagery time was 4 s, and the time interval between two motor imagery tasks was 1 s.

5. The application of the multi-person motion imagery recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: In step 1, a 64-electrode Neuroscan device was used to acquire EEG data with a sampling frequency of 1000 Hz.

6. The application of the multi-person motion imagination recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: In step 1, the motor imagery task is presented to the subjects through videos or pictures.

7. The application of the multi-person motion imagery recognition method based on cross-brain fusion decision in unmanned aerial vehicle control according to claim 1 is characterized in that: The preprocessing in step 2 includes bandpass filtering and de-averaging the reference.

8. A brain-computer system, used in the application of the multi-person motion imagery recognition method based on cross-brain fusion decision-making as claimed in claim 1 in unmanned aerial vehicle control, characterized in that: It includes a central processing unit PC1, a central processing unit PC2, a central processing unit PC3 and a central processing unit PC4; the central processing unit PC1 and the central processing unit PC2 are both provided with a multi-person EEG synchronous acquisition module and a preprocessing module, the multi-person EEG synchronous acquisition module acquires the user's EEG signal and transmits it to the preprocessing module for preprocessing; the central processing unit PC4 is provided with a task module and a label recognition module; the task module provides the trainee with a motor imagery task during training; the label recognition module generates a label; The central processing unit PC3 is provided with a data transmission and communication module, a feature calculation module, a multi-person fusion analysis and decision module, and a visualization module; the EEG signal output by the preprocessing module is labeled by the label recognition module and then output to the feature calculation module through the data transmission and communication module; The feature calculation module calculates the classification feature matrix of the labeled EEG data; The classifier in the multi-person EEG signal synchronous acquisition module is trained using the classification feature matrix generated by the feature calculation module, and after training, the motor imagery category is identified for the multi-person fused EEG signals, and the identification results are displayed on the visualization module.

Images