Collaborative Motor Imagery Decoding Method and Brain-Computer System Based on Hypergraph Representation

Abstract

The present invention discloses a collaborative motor imagery decoding method based on hypergraph representation, comprising the following steps: S1, preprocessing a previously collected multi-brain motor imagery dataset; S2, extracting interaction information between individuals from the data using a hypergraph representation method; S3, extracting and compressing the extracted interaction information to reduce the computational overhead of model training, and using the extracted and compressed features to train an STM model to obtain a classification model; S4, loading the trained classification model into an online brain-computer interface system; S5, collecting real user data through an EEG data acquisition device, and inputting the real data as test samples into the classification model in the online system to perform classification decisions and obtain recognition results. This is a multi-person online brain-computer system with high accuracy and stability for online recognition. The feature classification module uses a support tensor machine (STM) model to address the small sample size problem.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-pattern recognition-brain-computer interface technology, and specifically to a collaborative motor imagery decoding method based on hypergraph representation and an online brain-computer interface system thereof. Background Art

[0002] Motor imagery BCI is an active BCI technology based on electroencephalogram (EEG) signals. It can control external devices by capturing, analyzing, and classifying EEG signals generated by the human brain during motor imagery, without the need for actual physical movements. As a non-invasive, convenient, and safe rehabilitation method, motor imagery BCI has been widely used in the rehabilitation treatment of various physical injuries and movement disorders. For example, the use of motor imagery BCI technology in the rehabilitation treatment of Parkinson's patients can improve their walking gait and hand movement ability. However, existing active BCIs have technical bottlenecks: low decoding accuracy and weak stability.

[0003] The emergence of hyperscanning technology has broken the technical bottleneck of active BCI technology. Using EEG for research, hyperscanning is a technology that can synchronously record the brain activity of two or more users interacting under specific driving or cognitive tasks. Collaborative motor imagery brain-computer interfaces can be implemented using hyperscanning technology. This approach allows multiple users to collaborate on tasks, integrating and analyzing their respective EEG signals, thereby improving the performance of the BCI system. In collaborative motor imagery BCIs based on hyperscanning, the decoding problem primarily involves feature extraction and classification of EEG signals from multiple users. Decoding is key to improving the overall performance of the system.

[0004] However, current decoding methods used in multi-brain-computer interfaces (MBIs) primarily involve linear averaging or feature integration learning techniques for multi-brain EEG data. Research is lacking in exploring collaborative motor imagery BCIs for multiple individuals, or in effectively utilizing the interactive relationships between individuals during multi-brain activity to decode BCIs. Therefore, significant challenges remain in improving the decoding accuracy of collaborative motor imagery BCIs for multiple individuals. Summary of the Invention

[0005] The purpose of this invention is to solve the limitations of traditional collaborative motor imagery brain-computer interface decoding methods, explore the complex interactive relationships between different users when performing multi-person collaborative tasks, and based on hypergraph learning, describe high-order relationships between multiple nodes. A collaborative motor imagery decoding method and brain-computer system based on hypergraph representation are proposed. It is a multi-person online brain-computer system with high precision and high stability in online recognition. The feature classification module adopts the support tensor machine (STM) model to deal with the small sample problem.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A collaborative motor imagery decoding method based on hypergraph representation comprises the following steps:

[0008] S1. Preprocess the multi-brain motor imagery dataset collected in advance;

[0009] S2. Using hypergraph representation methods, extract interaction information between individuals from the data;

[0010] S3. The extracted interaction information is decomposed into core features and features compressed from three dimensions: user, channel, and channel, to reduce the computational overhead of model training. The features after core feature extraction and feature compression are then used to train the STM model to obtain a classification model.

[0011] S4, loading the trained classification model into the online brain-computer interface system;

[0012] S5. Collect the user's real data through the EEG data acquisition device, and input the real data as a test sample into the classification model in the online system to make classification decisions to obtain recognition results.

[0013] Preferably, the preprocessing method in step S1 includes: extracting EEG data in the 8-30 Hz frequency band from the data set by bandpass filtering; and removing ECG and artifacts caused by eye movement by blind source separation.

[0014] Preferably, in step S2, the method for extracting the interaction information between individuals is as follows:

[0015] Define the hypergraph as Y = (V, ε, W), which includes the vertex set V, the edge set ε, and the diagonal matrix W of edge weights. The hypergraph Y can be represented by the |V| × |ε| incidence matrix H, which is defined as

[0016]

[0017] H is obtained based on sparse linear regression. The specific formula is as follows:

[0018]

[0019] Among them A i =(x1,…,x i-1 ,x i+1 ,…,x N )∈R P×N-1 is the EEG signal of all channels after removing the i-th channel, α i ∈R N-1It is a coefficient vector that quantifies the influence of other channels on the i-th channel, and λ>0 is a regularization parameter that controls the sparsity of the solution.

[0020] Preferably, in step S3, the method of extracting and compressing features is:

[0021] The hypergraphs of multiple users are concatenated to obtain a three-dimensional tensor, which is subjected to Tucker decomposition to obtain the core tensor.

[0022] Preferably, the decomposition formula of the Tucker decomposition is as follows:

[0023]

[0024] Where X is the original tensor, A∈R I×P ,B∈R J×Q ,C∈R K×R It is a factor matrix, P, Q, R, p, q, r represent the number of columns in the corresponding factor matrix, g, a, b, c are the core tensor G and the corresponding factor tensors of the factor matrix A, B, C. The core tensor G represents the connection between each dimensional component and integrates multidimensional information.

[0025] Preferably, in step S3, the STM model is calculated using the following method:

[0026]

[0027] Where C represents the penalty coefficient, ξ represents the slack variable, the input parameter is the tensor χ, and the normal vector is in tensor form ω.

[0028] Preferably, the EEG data acquisition device is a Neuroscan device with 64-channel electrodes and a sampling frequency of 1000 Hz.

[0029] Preferably, the number of channels P selected for collecting EEG data is 62, and the 62 channels are FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO5, PO3, POz, PO4, PO6, PO8, O1, Oz, O2, CB1 and CB2 in the Neuroscan device.

[0030] The present invention also provides a brain-computer system for implementing the above-mentioned collaborative motor imagery decoding method based on hypergraph representation, which is characterized by including a multi-brain EEG existing data set preprocessing unit PU1, a multi-brain EEG signal decoding processing unit PU2 and a multi-brain brain-computer online system operation unit PU3.

[0031] The existing multi-brain EEG data set preprocessing unit PU1 is used to perform two preprocessing operations: bandpass filtering and blind source separation;

[0032] The multi-brain EEG signal decoding processing unit PU2 includes a multi-person EEG data feature extraction module and a classification model training module.

[0033] The multi-person EEG data feature extraction module is used to extract the interaction relationship between individuals in the same motor imagery task based on hypergraph learning, and then use tensor decomposition to extract and compress the core features before transmitting them to the classification model training module;

[0034] Classification model training module, used for offline classification model training;

[0035] The multi-brain brain-computer online system operation unit PU3 includes a user information registration module, a multi-brain EEG data synchronization acquisition module, and a pre-processing module.

[0036] The user information registration module is used to register the user information participating in the motor imagery task in preparation for the subsequent experimental data collection;

[0037] The multi-brain EEG data synchronization acquisition module is used to synchronize the hardware using a synchronization box, perform time calibration on the EEG signals of multiple participants to maintain synchronization, and then transmit them to the preprocessing module for bandpass filtering and blind source separation preprocessing.

[0038] The present invention has the following characteristics and beneficial effects:

[0039] The collaborative motor imagery brain-computer interface decoding method based on hypergraph representation proposed in the present invention solves the problem of extracting and decoding the interaction relationship between individuals in the collaborative motor imagery brain-computer interface system. It uses hypergraph to better represent the interaction relationship between individuals, and uses tensor decomposition technology to extract and compress the interpretable features therein, which can significantly improve the classification accuracy of motor imagery tasks.

[0040] This invention has the capabilities of collecting multi-user, multi-channel EEG data, extracting interactive relationship features, and visualizing classification results. It differs from traditional motor imagery brain-computer interfaces in the following ways: It simultaneously records the brain activity of two or more users interacting under specific driving or cognitive tasks, thereby revealing the brain-brain mechanisms of social interaction; it characterizes the interactive relationships between users and extracts core features, achieving high classification accuracy; and, given the long offline calibration time and insufficient training samples for active brain-computer interfaces, which can easily overfit the classifier, a classifier for small samples has been designed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a system framework diagram of the present invention.

[0043] Figure 2 It is a system flow chart of the present invention.

[0044] Figure 3 This is a channel diagram for collecting EEG data of the present invention.

[0045] Figure 4 This is a schematic diagram of the decoding method of the multi-person online brain-computer interface system of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0048] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0049] This embodiment takes into account the long calibration time of offline experiments and the difficulty of collecting large amounts of high-quality EEG data due to the user's tendency to be distracted during real-time experiments. Therefore, we aim to achieve high recognition accuracy for motor imagery tasks even with a small sample size. This system is a multi-person online brain-computer system with high accuracy and stability for online recognition. The feature classification module uses a support tensor machine (STM) model to address the small sample size issue.

[0050] Specifically, this embodiment provides a collaborative motor imagery decoding brain-computer interface based on hypergraph representation, such as Figure 2 As shown, it includes processing units PU1, PU2 and PU3.

[0051] The existing multi-brain EEG data set preprocessing unit PU1 includes two preprocessing methods: bandpass filtering and blind source separation.

[0052] The multi-brain EEG signal decoding and processing unit PU2 includes a multi-person EEG data feature extraction module and a classification model training module. The multi-person EEG data feature extraction module uses hypergraph learning to extract the interaction relationship between individuals in the same motor imagery task. It then uses tensor decomposition to extract and compress core features, which are then transmitted to the classification model training module for offline classification model training.

[0053] The multi-brain BCI online system operating unit PU3 includes a user information registration module, a multi-brain EEG data synchronization acquisition module, and a preprocessing module. The user information registration module registers the information of users participating in the motor imagery task, preparing for subsequent experimental data collection. The multi-brain EEG data synchronization acquisition module uses a synchronization box for hardware synchronization, aligning the EEG signals of multiple participants to maintain synchronization, and then transmitting them to the preprocessing module for bandpass filtering and blind source separation preprocessing.

[0054] Finally, the online data obtained from PU3 is input into the trained classification model obtained from PU2 to obtain the online recognition result.

[0055] Specifically, this embodiment provides a collaborative motor imagery decoding method based on hypergraph representation, such as Figure 1 and Figure 4 As shown, the following steps are included:

[0056] Step S1: pre-process the multi-brain motor imagery dataset collected in advance to reduce the interference of artifacts.

[0057] The specific process includes: (1) designing a bandpass filter to extract EEG data in the 8-30 Hz frequency band from the collected data set; and (2) blind source separation to remove EEG and artifacts caused by eye movements.

[0058] Step S2: Using a hypergraph representation method to extract interaction information between individuals from the data, specifically including:

[0059] The interaction relationship between individuals based on hypergraph representation is represented as follows: the hypergraph is defined as Y = (V, ε, W), which includes the vertex set V, the edge set ε, and the diagonal matrix W of edge weights. The hypergraph Y can be represented by the |V| × |ε| association matrix H, which is defined as

[0060]

[0061] H is obtained based on sparse linear regression (i.e. Lasso). The specific formula is as follows:

[0062]

[0063] Among them A i =(x1,…,x i-1 ,x i+1 ,…,x N )∈R P×N-1 is the EEG signal of all channels after removing the i-th channel, α i ∈R N-1It is a coefficient vector that quantifies the influence of other channels on the i-th channel, and λ>0 is a regularization parameter that controls the sparsity of the solution.

[0064] Step S3: Use tensor decomposition to extract core features and compress features from three dimensions: user, channel, and channel, and use the features after core feature extraction and feature compression to train the STM model to obtain a classification model.

[0065] Specifically, the hypergraph representations of multiple users are concatenated to obtain a three-dimensional tensor, which is then subjected to Tucker decomposition to obtain the core tensor, as follows:

[0066] The decomposition formula of Tucker decomposition is as follows:

[0067]

[0068] Where X is the original tensor, A∈R I×P ,B∈R J×Q ,C∈R K×R It is a factor matrix, P, Q, R, p, q, r represent the number of columns in the corresponding factor matrix, g, a, b, c are the core tensor G and the corresponding factor tensors of the factor matrix A, B, C. The core tensor G represents the connection between each dimensional component and integrates multidimensional information.

[0069] Among them, the classification model selects the STM model suitable for the field of brain-computer interface, and the calculation method is as follows:

[0070]

[0071] Where C represents the penalty coefficient, ξ represents the slack variable, the input parameter is the tensor χ, and the normal vector is in tensor form ω. Because the STM model relies only on a small number of support tensors to obtain the optimal solution, the algorithm has good robustness and generalization ability, making it particularly suitable for solving small sample problems.

[0072] Step S4: Load the trained classification model into the online system.

[0073] Step S5, user information registration module: register the user information participating in the motor imagery task on the online system to prepare for the subsequent experimental data collection.

[0074] Step S6: The EEG data is collected by the multi-brain EEG data synchronization acquisition module using a Neuroscan device with 64 channels of electrodes and a sampling frequency of 1000 Hz. The synchronization control uses a synchronization box for hardware synchronization. The EEG signals of multiple participants are time-calibrated to maintain synchronization. The EEG cap adopts the international standard 10 / 20 system electrode placement method, such as Figure 3As shown in Figure 1, the 62 electrodes used are FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO5, PO3, POz, PO4, PO6, PO8, O1, Oz, O2, CB1, and CB2. The users sat side by side and simultaneously watched pictures of motor imagery task instructions to perform the motor imagery experiment. Based on different motor imagery types, the EEG data of each user was recorded and labeled accordingly.

[0075] Step S7: Preprocess the collected EEG data using the multi-brain EEG online data acquisition preprocessing module to reduce artifact interference. The specific process includes: (1) designing a bandpass filter to extract EEG data in the 8-30 Hz frequency band from the collected data set; and (2) performing blind source separation to remove ECG and artifacts caused by eye movements.

[0076] Step S7: Input the online data obtained from PU3 into the trained classification model obtained from PU2 to obtain an online recognition result, and determine the motor imagery task types of the two training users.

[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A collaborative motor imagery decoding method based on hypergraph representation, characterized in that: The steps include: S1. Preprocess the multi-brain motor imagery dataset collected in advance; S2. Using hypergraph representation methods, extract interaction information between individuals from the data; S3. The extracted interaction information is decomposed into core features and features compressed from three dimensions: user, channel, and channel. The features after core feature extraction and feature compression are trained on the STM model to obtain a classification model. S4, loading the trained classification model into the online brain-computer interface system; S5. Collect the user's real data through the EEG data acquisition device, and input the real data as a test sample into the classification model in the online system to make classification decisions to obtain recognition results.

2. The collaborative motor imagery decoding method based on hypergraph representation according to claim 1, characterized in that: The pre-processing method in step S1 includes: extracting EEG data in the 8-30 Hz frequency band from the data set through band-pass filtering; and removing electrocardiogram and artifacts caused by eye movement through blind source separation.

3. The collaborative motor imagery decoding method based on hypergraph representation according to claim 2, characterized in that: In step S2, the method for extracting the interaction information between individuals is as follows: Define the hypergraph as Y = (V, ε, W), which includes the vertex set V, the edge set ε, and the diagonal matrix W of edge weights. The hypergraph Y can be represented by the |V| × |ε| incidence matrix H, which is defined as H is obtained based on sparse linear regression. The specific formula is as follows: Among them A i =(x1,…,x i-1 ,x i+1 ,…,x N )∈R P×N-1 is the EEG signal of all channels after removing the i-th channel, α i ∈R N-1 It is a coefficient vector that quantifies the influence of other channels on the i-th channel, and λ>0 is a regularization parameter that controls the sparsity of the solution.

4. The collaborative motor imagery decoding method based on hypergraph representation according to claim 2, characterized in that: In step S3, the method for extracting and compressing core features is as follows: The hypergraphs of multiple users are concatenated to obtain a three-dimensional tensor, which is subjected to Tucker decomposition to obtain the core tensor.

5. The collaborative motor imagery decoding method based on hypergraph representation according to claim 4, characterized in that: The decomposition formula of the Tucker decomposition is as follows: Where X is the original tensor, A∈R I×P ,B∈R J×Q ,C∈R K×R It is a factor matrix, P, Q, R, p, q, r represent the number of columns in the corresponding factor matrix, g, a, b, c are the core tensor G and the corresponding factor tensors of the factor matrix A, B, C. The core tensor G represents the connection between each dimensional component and integrates multidimensional information.

6. The collaborative motor imagery decoding method based on hypergraph representation according to claim 1, characterized in that: In step S3, the STM model is calculated using the following method: Where C represents the penalty coefficient, ξ represents the slack variable, the input parameter is the tensor χ, and the normal vector is in tensor form ω.

7. The collaborative motor imagery decoding method based on hypergraph representation according to claim 1, characterized in that: The EEG data acquisition device is a Neuroscan device with 64-channel electrodes and a sampling frequency of 1000 Hz.

8. The collaborative motor imagery decoding method based on hypergraph representation according to claim 7, characterized in that: The number of channels P selected for collecting EEG data is 62. The 62 channels are FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO5, PO3, POz, PO4, PO6, PO8, O1, Oz, O2, CB1 and CB2 in the Neuroscan device.

9. A brain-computer system for implementing the collaborative motor imagery decoding method based on hypergraph representation according to any one of claims 1 to 8, characterized in that: It includes a multi-brain EEG existing data set pre-processing unit PU1, a multi-brain EEG signal decoding processing unit PU2 and a multi-brain brain-computer online system operation unit PU3; The existing multi-brain EEG data set preprocessing unit PU1 is used to perform two preprocessing operations: bandpass filtering and blind source separation; The multi-brain EEG signal decoding processing unit PU2 includes a multi-person EEG data feature extraction module and a classification model training module. The multi-person EEG data feature extraction module is used to extract the interaction relationship between individuals in the same motor imagery task based on hypergraph learning, and then use tensor decomposition to extract and compress the core features before transmitting them to the classification model training module; Classification model training module, used for offline classification model training; The multi-brain brain-computer online system operation unit PU3 includes a user information registration module, a multi-brain EEG data synchronization acquisition module, and a pre-processing module. The user information registration module is used to register the user information participating in the motor imagery task in preparation for the subsequent experimental data collection; The multi-brain EEG data synchronization acquisition module is used to synchronize the hardware using a synchronization box, perform time calibration on the EEG signals of multiple participants to maintain synchronization, and then transmit them to the preprocessing module for bandpass filtering and blind source separation preprocessing.

Images