Group motor imagery brain-computer interface decoding method and system based on feature interaction

Abstract

The application discloses a kind of group motor imagery brain-computer interface decoding methods based on feature interaction, comprising the following steps: S1, the same motor imagery task is provided to multiple users simultaneously;S2, signal acquisition, before brain-computer interface decoding, the continuous electroencephalogram of all users needs to be simultaneously collected;S3, the electroencephalogram signal of acquisition is preprocessed, and the electroencephalogram signal network is obtained by calculating the adjacent matrix and node feature of the electroencephalogram signal after preprocessing;S4, the feature of each user's electroencephalogram signal network is extracted using graph convolution network;S5, feature interaction, the feature vector of current user is fused with the feature vector of other users using the technology of variational autoencoder, to obtain the feature with cross-brain interaction information.S6, the interactive feature with label is trained using a classifier and the result is output.The method considers the current state of user, dynamically adjusts the weight of different users, and enhances the accuracy of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence-pattern recognition-brain-computer interface, and in particular to a group motor imagery brain-computer interface decoding method and system based on feature interaction. BACKGROUND

[0002] A brain-computer interface (BCI) is a technology that connects the human brain to the outside world by directly controlling computers or other devices. It uses biological signals such as brain waves, brain magnetic waves, and cerebrospinal fluid to convey human intentions and thoughts, thereby controlling the operation of external devices. With the continuous progress of technology and the expansion of application scenarios, brain-computer interface technology is expected to become an important tool to help humans better control and understand the world around us. Motor imagery (MI) refers to the generation of similar neural activity to actual movement by imagining a certain movement or action. Motor imagery is considered a very promising brain-computer interface technology that can enable humans to control external devices such as prosthetics, wheelchairs, and robots through brain signals. Group motor imagery brain-computer interface is a technology that records the electroencephalogram signals of multiple people simultaneously to achieve group collaborative control of external devices. Compared to individual motor imagery brain-computer interface, group motor imagery brain-computer interface allows multiple people to jointly control the same device to achieve more complex tasks.

[0003] Although group motor imagery brain-computer interface has many potential applications, such as in group collaboration and interactive games, it still has the following deficiencies and challenges:

[0004] 1. Group motor imagery brain-computer interface technology has certain difficulties in identifying and classifying a large number of electroencephalogram signals. Because electroencephalogram signals are affected by many factors such as muscle movement and environmental noise, efficient and accurate algorithms are needed to separate and classify signals from multiple people.

[0005] 2. Real-time performance and stability of group motor imagery brain-computer interface technology is also a challenge. When multiple people perform motor imagery simultaneously, because each person's brain activity pattern and speed may be different, real-time algorithms are needed to process signals and ensure the stability and reliability of the entire system.

[0006] 3. Group motor imagery task is actually a social behavior, and existing analysis strategies are still based on the transfer of single-brain analysis, without focusing on the essence of multi-brain data, i.e., information interaction between brain and brain. This new system can provide more natural human-computer interaction for users. SUMMARY

[0007] The present application aims at the defects and challenges of the current group motor imagery brain-computer interface, and provides a group motor imagery brain-computer interface decoding method and system based on feature interaction, which considers the current state of the user, dynamically adjusts the weight of different users, and enhances the accuracy of the system.

[0008] To solve the above technical problems, the technical scheme of the present application is:

[0009] A group motor imagery brain-computer interface decoding method based on feature interaction, comprising the following steps:

[0010] S1, the same motor imagery task is provided to multiple users at the same time, and the motor imagery task contains multiple motor imagery states (left hand, right hand, idle state, etc.);

[0011] S2, signal acquisition, before brain-computer interface decoding, the continuous electroencephalogram signals of all users need to be collected at the same time. A hat or headband with electrodes is generally used to collect signals, which are amplified and converted into digital signals for subsequent processing. Each user collects the prefrontal part and central area related to motor imagery.

[0012] S3, preprocessing, after signal acquisition, the signal of each user needs to be preprocessed to remove noise and improve signal quality. This usually includes removing noise caused by eye movement and muscle movement, filtering to remove low and high frequency components, and baseline calibration to remove basic signal offset, and calculating the adjacency matrix and node features of each subject for each trial to obtain the electroencephalogram signal network.

[0013] S4, feature extraction, using graph convolutional network (GCN) to extract features from each user to convert the original signal into a feature vector that can be used by machine learning algorithms, which can help identify the electroencephalogram signal when the user is motor imagining;

[0014] S5, feature interaction, the feature vector of the current user is fused with the feature vector of other users using the technology of variational autoencoder (VAE), to obtain a feature with cross-brain interaction information.

[0015] S6, classifier training, the labeled interactive features are trained by the classifier and the results are output.

[0016] As a preferred, the duration of each motor imagery task in step S1 is 4 seconds, wherein the first second plays a specific task video, and the last 3 seconds perform the motor imagery task.

[0017] Preferably, in step S2, the signal acquisition device uses a 64-lead device of Neuroscan, and the electrode placement position refers to the international 10-20 standard lead system, and specifically uses 8 electrodes in the forehead region: Fp1, Fp2, Fpz, AF7, AF3, AFz, AF4, AF8, and 7 electrodes in the central region: C5, C3, C1, Cz, C2, C4, C6, a total of 15 electrodes.

[0018] Preferably, in step S2, the synchronous acquisition mode is that the client sends a synchronization tag to all acquisition devices at the same time, and the synchronization is performed according to the synchronization tag in the received electroencephalogram information.

[0019] Preferably, in step S4, the graph convolution network algorithm includes two graph convolution layers, two graph pooling layers and three fully connected layers. The graph construction mode quantitatively describes the correlation degree of two channels by using a correlation coefficient, that is, the greater the correlation coefficient index, the greater the correlation degree.

[0020] The application also provides a feature interaction-based group motor imagery brain-computer interface decoding system, which comprises (1) PC1: a central processing unit that schedules hardware resources and manages other processing units; (2) PC2: a signal receiving unit that receives and buffers electroencephalogram data from each user and receives synchronization information from PC4; (3) PC3: an offline training processing unit that provides an initial and trained model to the system; (4) PC4: a signal synchronization unit that is responsible for sending a synchronization tag to a serial port; (5) PC5: a decoding unit that is a processing unit for the specific implementation of the feature interaction-based group motor imagery brain-computer interface decoding method; (6) PC6: a fatigue detection unit; and (7) PC7: an output unit that collects results and fatigue states and provides an interface to the outside.

[0021] In the decoding system, the main interaction behaviors between the processing units are as follows: PC1 starts the corresponding processing unit according to the offline or online mode selected by the user. When the user selects the offline mode, PC1 loads offline data, starts PC3, PC5, PC6 and PC7, and after the initialization of each module, PC3 sends a pre-trained method model to PC5, PC5 starts to decode the signal, and PC6 synchronously calculates the fatigue index. After receiving the processing results of PC5 and PC6, PC7 outputs the processing results to the outside.

[0022] When the user selects the online mode, PC1 first promotes the priority of PC2 to the highest level, and starts all other modules. PC3 provides an initial decoding model. After the modules are initialized, PC2 starts to receive data. PC4 sends a synchronization signal to the acquisition device, and PC4 enters a dormant state after confirming successful sending. After receiving the synchronization information, PC2 starts to synchronize the signal. After synchronization is completed, PC2 starts to transmit data to PC5 and PC6, and PC5 and PC6 send the results to PC7 after processing the data, and update the model of PC3.

[0023] The application has the following characteristics and beneficial effects:

[0024] The group motor imagery brain-computer interface decoding method based on feature interaction provided by the application can synchronize the electroencephalogram data of multiple users through an electroencephalogram device, extract the relationship with the interaction behavior of other users, and fuse the interaction features of other users on the basis of the own feature vector, so that the recognition accuracy of the group motor imagery task can be significantly improved.

[0025] The group motor imagery brain-computer interface system based on feature interaction provided by the application has the functions of synchronous recording of signals of multiple multi-channel electroencephalogram acquisition devices, extraction of interaction relationship features in a task state, classification and identification, etc. The difference from the single brain-computer interface is that the application adds information of interaction with other users on the basis of a single brain. These additional information is often related to the current task, so the accuracy is higher than that of the single brain mode. The data synchronization problem of multiple devices is solved. In the group brain-computer interface, multiple people participate at the same time, and the electroencephalogram signals of them are calibrated in time, so that the signals are collected and processed at the same time. In this way, the time delay and interference between different signals can be avoided, the signals of different people have the same time reference system, and the merging and classification of the signals are facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, below the drawings needed to be used in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 The system framework diagram of the application.

[0028] Figure 2 The decoding method flowchart of the application.

[0029] Figure 3 The electroencephalogram recording channel diagram of the application.

[0030] Figure 4 Figure 1 is a schematic diagram of a multi-person brain-computer interface system according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0032] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0033] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0034] The present embodiment provides a group motor imagery brain-computer interface decoding method based on feature interaction, as shown in Figure 1 and Figure 2 The method comprises the following steps:

[0035] S1, the same motor imagery task is provided to multiple users at the same time, and the motor imagery task comprises multiple motor imagery states (left hand, right hand, idle state, etc.);

[0036] S2, signal acquisition, before brain-computer interface decoding, the continuous electroencephalogram signals of all users need to be collected at the same time. A cap or headband with electrodes is generally used to collect signals, which are amplified and converted into digital signals for subsequent processing. Each user collects the prefrontal part and central area related to motor imagery.

[0037] In this embodiment, the signal acquisition device uses Neuroscan's 64-lead device, and the electrode placement position refers to the international 10-20 standard lead system. Specifically, 8 electrodes in the forehead region are used: Fp1, Fp2, Fpz, AF7, AF3, AFz, AF4, AF8, and 7 electrodes in the central region: C5, C3, C1, Cz, C2, C4, C6, a total of 15 electrodes.

[0038] It should be noted that the synchronous acquisition mode is that the client sends a synchronization tag to all acquisition devices at the same time, and the synchronization is performed according to the synchronization tag in the received electroencephalogram information.

[0039] S3, preprocessing stage: first, a Butterworth wireless pulse filter is used to denoise and filter the signal; the baseline calibration is performed on the signal in the form of task state signal minus the average value of resting state signal to solve the signal offset problem.

[0040] Then, this part is to calculate the adjacency matrix and node features for each trial of each subject. At present, the value of the adjacency matrix is calculated based on the Pearson correlation coefficient (PCC) based on correlation. The correlation coefficient quantitatively describes the correlation degree of i and j channels. The Pearson correlation coefficient based on correlation constructs an undirected multi-brain network as shown in formula (1):

[0041]

[0042] The node feature is a two-dimensional feature composed of the degree of the node and the differential entropy (DE). The degree of the node is the sum of the row or column after the binary of the above adjacency matrix. Finally, the adjacency matrix and the node feature constitute the electroencephalogram signal network network, and then the network features are extracted.

[0043] S4, extracting network features: this stage uses a graph convolution network GCN to extract a high-dimensional and low-dimensional feature vector for each electroencephalogram signal network network, respectively. GCN is a neural network capable of extracting graph (electroencephalogram signal network network) features. The high-dimensional vector is input into the VAE network of stage 3 for classification, and the low-dimensional vector is input into the VAE network of other users for feature fusion.

[0044] Stage 3 network feature fusion: in this stage, the VAE network reduces the dimension of the high-dimensional feature to match the dimension of the low-dimensional feature, and receives the low-dimensional feature from other subjects. The mapping of each feature vector in the low dimension obeys the normal distribution, so that the extracted features have stronger stability. In order to measure the similarity between the current trained distribution and the normal distribution, the KL divergence is used to constrain the network, and the definition formula (2) of the KL divergence is shown.

[0045]

[0046] where p is the standard normal distribution, q is the distribution fitted by the neural network, and x is the sample point. Then the loss function of the VAE network is defined as shown in equation (3).

[0047]

[0048] where f i is the i-th element of the input high-dimensional feature vector, is the output result of the AVE network decoder. The low-dimensional features of other subjects are fused with the low-dimensional representation of the high-dimensional features through the bottleneck layer of the VAE. During the training process of the VAE, the VAE automatically assigns weights to each value of the low-dimensional vector, focuses on learning the values conducive to classification, and weakens the values not conducive to classification. Finally, the fused vector is input into an artificial neural network (ANN) classifier. Therefore, the final loss function of the network in stage 3 is shown in equation (4).

[0049]

[0050] The embodiment also provides a group motor imagery brain-computer interface system based on feature interaction, as shown in Figure 4 The system has the seven main processing units described above:

[0051] PC1: The central processing unit is responsible for the opening, pausing and stopping of the system and the scheduling of system resources. It includes a resource management module, a data transmission module and a unit control module. The resource management module allocates the number and priority of processes for other execution units; the unit control module is a module common to all modules and is responsible for communication between itself and other processing units; and the data transmission module is responsible for receiving and forwarding offline data.

[0052] PC2: The signal receiving processing unit is responsible for receiving, buffering, synchronizing and forwarding data. It includes a unit control module, a data buffering module, a synchronization module and a data transmission module. Before signal synchronization, a buffer is needed to buffer the signals due to the continuous arrival of the collected data. After receiving the synchronization signal from PC4, the signals are synchronized according to the synchronization markers of each user. Finally, the data transmission module forwards the data.

[0053] PC3: The offline training processing unit provides an initial and trained model, and transmits the stored model or receives an updated model through a model transmission module. It includes a unit control module, a model storage module and a model transmission module.

[0054] PC4: The signal synchronization processing unit receives the control signal of PC2 to send a synchronization label to the serial port, and the label is punched into the electroencephalogram of each user. It includes a unit control module and a serial port transmission module.

[0055] PC5: decoding processing unit is responsible for decoding of signals, including unit control, decoding and data transmission module. The data transmission module of the unit receives data from PC2, and the decoding module is responsible for outputting the decoding result, and the result is sent to PC7 through the data transmission module.

[0056] PC6: fatigue detection processing unit is responsible for identifying the current fatigue state, including unit control, fatigue identification and data transmission module. The data transmission module of the unit receives data from PC2, and the fatigue identification is responsible for outputting the fatigue state of each user, and the result is sent to PC7 through the data transmission module.

[0057] The fatigue state uses the energy ratio of each frequency band in the electroencephalogram signal as an indicator. The specific operation is as follows:

[0058] 1. Design a band-pass filter for 4-8 Hz, 8-13 Hz, and 13-30 Hz, respectively.

[0059] 2. Use the band-pass filter to filter the electroencephalogram signal, and then square and process the signal to obtain the energy of three different frequency bands, denoted as θ, α and β, respectively.

[0060] 3. Calculate the following fatigue values: (α+θ) / β, α / β, (α+θ) / (α+β) and θ / β, respectively.

[0061] 4. Add the four fatigue values obtained in step 3 to obtain the final fatigue indicator.

[0062] PC7: output processing unit receives and stores the decoding results and fatigue states of PC5 and PC6, and transmits to external output devices. The module is composed of unit control, result storage and data transmission module.

[0063] In addition, the external devices used for the case of 2 users in the example are: two sets of Neuroscan 64-channel electroencephalogram acquisition devices, one display screen for prompting motor imagination tasks, and one display screen for result output.

[0064] After starting each processing unit, PC1 sends offline data to PC5 and PC6 through the data transmission module in offline mode, and PC3 sends the pre-trained method model to PC5. PC5 starts decoding the signal after receiving the model, and sends the structure to PC7 after decoding. PC6 receives offline data and calculates the fatigue indicator for each subject. PC7 receives the processing results of PC5 and PC6 and outputs the processing results to the external output device;

[0065] When the user selects the online mode, in order to receive online data in time, PC1 first promotes the priority of PC2 to the highest level, and then initializes other modules. PC3 transmits the initial decoding model to PC5. PC2 starts to receive and buffer data, and sends a synchronization control signal to PC4, and then enters a waiting state until receiving a signal of the synchronization tag. This is the synchronization module starting to synchronize signals according to the synchronization tag. After synchronization is completed, PC2 starts to transmit data to PC5 and PC6, PC5 starts decoding after receiving the model and data, and sends the result to PC7. PC6 sends the result to PC7 after identifying the fatigue state, and updates the model of PC3. PC7 receives the decoding result and the fatigue state, saves them and forwards them to an external device.

[0066] To prove the advantages of the embodiment compared with the prior art, we compared the recognition accuracy of 8 groups of two-person motor imagery tasks, as shown in Table 1. The comparison experiment classifies the node features by support vector machine (SVM), and the features extracted by GCN are classified by the ANN classifier in this project (without feature embedding, denoted as non-fusion; after embedding, denoted as fusion), and the final result is the result after 10-fold cross-validation. It can be proved that the interaction mode is better than the performance of each subject in the non-interaction mode.

[0067]

[0068] The embodiments of the application are described in detail above with reference to the drawings, but the application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components can be made without departing from the principles and spirits of the application, and still fall within the protection scope of the application.

Claims

1. A feature interaction-based group motor imagery brain-computer interface decoding method, characterized in that, The steps are as follows: S1, providing the same motor imagery task to multiple users at the same time; S2, collecting continuous electroencephalogram signals of all users through an electroencephalogram signal collector; S3, preprocessing the collected electroencephalogram signals, and calculating the adjacency matrix and node features of the preprocessed electroencephalogram signals to obtain an electroencephalogram signal network; The value of the adjacency matrix is calculated based on the Pearson correlation coefficient based on correlation, which quantitatively describes the correlation degree of the i and j channels, and the Pearson correlation coefficient based on correlation is used to construct the adjacency matrix. The node feature is a two-dimensional feature composed of the degree of the node and the differential entropy, and the degree of the node is the sum of the rows or columns after binarization of the adjacency matrix. Finally, the adjacency matrix and the node feature constitute the electroencephalogram signal network. S4, using a graph convolution network to extract features of the electroencephalogram signal network of each user, and converting them into high-dimensional feature vectors and low-dimensional feature vectors for machine learning algorithms; S5, reducing the dimension of the high-dimensional feature vector to match the dimension of the low-dimensional feature vector, and receiving low-dimensional features from other subjects.

2. The feature interaction-based group motor imagery brain-computer interface decoding method according to claim 1, wherein, The variational autoencoder makes the mapping of each feature vector in low dimension subject to a normal distribution, and the low-dimensional feature vectors of other subjects are fused with the low-dimensional representation of the high-dimensional feature vector through the bottleneck layer of the variational autoencoder to obtain a cross-brain interaction information feature; 3. The feature interaction-based group motor imagery brain-computer interface decoding method according to claim 1, wherein, S6, using a classifier to train the labeled interaction information feature and outputting the result.

4. The feature interaction-based group motor imagery brain-computer interface decoding method according to claim 3, characterized in that, In step S1, each motor imagery task lasts for 4 seconds, including 1 second of playing a specific task video and 3 seconds of motor imagery task.

5. The feature interaction-based group motor imagery brain-computer interface decoding method according to claim 4, wherein, In step S2, the forehead and central region of each user are collected by the electroencephalogram signal collector.

6. The feature interaction-based group motor imagery brain-computer interface decoding method according to claim 5, wherein, The electroencephalogram signal collector uses a Neuroscan 64-channel device, specifically using 8 electrodes in the forehead region: Fp1, Fp2, Fpz, AF7, AF3, AFz, AF4, AF8, and 7 electrodes in the central region: C5, C3, C1, Cz, C2, C4, C6, a total of 15 electrodes.

7. The feature interaction based group motor imagery brain-computer interface decoding method according to claim 1, wherein, In step S2, the electroencephalogram signals of the users are collected by synchronous acquisition.

8. The feature interaction based group motor imagery brain-computer interface decoding method according to claim 1, wherein, The synchronous acquisition method is to send a synchronization tag to all Neuroscan 64-channel devices simultaneously through the client, and synchronize according to the synchronization tag in the received electroencephalogram information.

9. A system for implementing the feature interaction-based group motor imagery brain-computer interface decoding method of any one of claims 1-8, characterized in that, In step S3, the preprocessing method of the collected electroencephalogram signals is to remove noise caused by eye movement and muscle movement, filter to remove low and high frequency components, and baseline calibration to remove basic signal offset. In step S4, the graph convolution network includes two graph convolution layers, two graph pooling layers, and three fully connected layers. The correlation coefficient quantitatively describes the correlation degree of two channels, that is, the larger the correlation coefficient index, the greater the correlation degree. It includes: PC1: central processing unit, scheduling hardware resources and managing other processing units; PC2: signal receiving unit, receiving and buffering electroencephalogram data from each user, and receiving synchronization information from PC4; PC3: offline training processing unit, providing an initial and trained model to the system; PC4: signal synchronization unit, responsible for sending synchronization tags to the serial port; PC5: decoding unit, a processing unit based on the specific implementation of the feature interaction group motor imagery brain-computer interface decoding method; PC6: fatigue detection unit; PC7: output unit, collects results and fatigue state and provides an interface to the outside.

Images