Cross-individual brain-computer interface system based on convolutional neural networks and EEG big data

Abstract

The present invention discloses a cross-individual brain-computer interface system based on convolutional neural network and EEG big data, including: a stimulus presentation module, presenting visual stimulation through a graphical user interface to induce the user's P300 potential; a signal acquisition module, collecting the user's EEG signal; a signal preprocessing and feature extraction module, preprocessing and feature extraction of the EEG signal; a classifier establishment and pre-training module, using the collected large-scale EEG data set to optimize model parameters, fitting a cross-individual P300 detection pre-training model; a classifier fine-tuning module, using a small number of new user samples to fine-tune the cross-individual P300 detection pre-training model, fitting a cross-individual P300 detection fine-tuning model that is more suitable for a specific new user; an online prediction module, using the cross-individual P300 detection pre-training model or the cross-individual P300 detection fine-tuning model to predict and analyze the pre-processed samples during the online manipulation process, and output the results. The present invention can shorten or even completely remove the calibration stage, and improve the practicality and ease of use of the brain-computer interface.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical signal processing, machine learning algorithms and brain-computer interfaces, and in particular to a cross-individual brain-computer interface (BCI) system based on convolutional neural network (CNN) and scalp electroencephalography (EEG) big data. Background Art

[0002] Brain-computer interfaces measure and analyze brain activity and convert this activity into computer commands to control external devices. Brain-computer interface technology provides a new type of human-computer interaction and also facilitates elderly care and assistance for the disabled. Through brain-computer interfaces, people with motor impairments can achieve functions such as text input, control wheelchairs, and household appliances without physical movement. Because EEG has an extremely low signal-to-noise ratio and strong non-stationarity, and individual differences are large, operating traditional EEG-based BCI systems usually requires a calibration phase. In this phase, users need to perform a sufficient amount of specific tasks according to the instructions of the BCI so that the BCI can collect training data sets and use the training data sets to establish a machine learning model for specific users for prediction during online control. Not only new users need calibration, but old users also need to recalibrate during long-term use or after changing to different models of EEG devices. However, calibration usually takes 10-20 minutes, or even longer, and requiring users to complete calibration before each use is usually time-consuming and inconvenient. Therefore, it is particularly important to shorten or even completely remove the calibration phase to improve user experience. However, developing BCIs with satisfactory performance across individuals remains a challenge, mainly because EEG patterns are highly individual-specific, dynamic, and have a low signal-to-noise ratio. In addition, multi-center (multi-device) data collection and analysis of EEG big data have not yet been invented.

[0003] Deep neural networks are usually composed of multiple layers to extract multi-level hierarchical structures of features and combine feature extraction with regression or classification. CNN is one of the most commonly used models in deep learning. CNN usually consists of one or more convolutional layers, pooling layers, and a fully connected layer at the top. This structure enables convolutional neural networks to extract high-level local and global features using the two-dimensional structure of the input data.

[0004] Transfer learning is a hot research topic in the field of machine learning. Given a source domain and a source task, a target domain and a target task, the goal of transfer learning is to use the knowledge learned from the source domain and source task to improve the performance of the classifier on the target domain and target task when the source domain and target domain are different or the source task and target task are different. When the number of training samples in the target domain and target task is insufficient, but there are sufficient training samples in other related domains, transfer learning can often greatly improve the performance of the classifier on the target domain and target task. When using a deep neural network as a classifier, pre-training the classifier with samples from the source domain and then fine-tuning the classifier with a small amount of samples from the target domain is a simple and effective transfer learning method.

[0005] Electroencephalography (EEG) is an electrophysiological monitoring method that records brain waves non-invasively. Using EEG-based BCI often requires users to wear an electrode cap with non-invasive electrodes arranged according to certain standards, and inject conductive glue between the electrodes and the scalp to reduce the contact impedance between the scalp and the electrodes, facilitating signal acquisition. Compared with other brain signal measurement technologies, EEG acquisition has the advantages of being non-invasive, convenient, and low-cost, and therefore is widely used.

[0006] The P300 potential is a common event-related potential (ERP) component. It is considered an endogenous potential because its occurrence is not related to the physical properties of the stimulus, but to the person's response. P300 is usually induced using the oddball paradigm, that is, low-probability targets are mixed with high-probability non-targets, and the potential is induced by low-probability target stimulation. It is manifested in EEG recordings as a positive deflection of the potential approximately 300ms after the stimulus appears. Although EEG is highly complex and has an extremely low signal-to-noise ratio, the repeatability and ubiquity of this signal make it a common choice for clinical and laboratory testing, and also a common choice for BCI systems with stable performance and easy operation. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing P300 brain-computer interface that requires a time-consuming and labor-intensive calibration process, and proposes a cross-individual brain-computer interface system based on convolutional neural networks and EEG big data, which can greatly shorten or even completely eliminate the calibration stage, thereby improving the practicality and ease of use of the brain-computer interface.

[0008] To achieve the above purpose, the technical solution provided by the present invention is: a cross-individual brain-computer interface system based on convolutional neural network and EEG big data, comprising:

[0009] A stimulus presentation module presents visual stimuli through a graphical user interface to induce the user's P300 potential;

[0010] A signal acquisition module, used to collect the user's EEG signal;

[0011] A signal preprocessing and feature extraction module is used to preprocess the EEG signal, including removing noise and baseline, and extracting features from the preprocessed signal;

[0012] The classifier building and pre-training module is used to build a convolutional neural network model, namely a CNN model, and optimize the model parameters using a large-scale EEG data set collected from a large number of subjects, and fit a cross-individual P300 detection pre-training model based on a convolutional neural network;

[0013] The classifier fine-tuning module uses a small number of new user samples to fine-tune the cross-individual P300 detection pre-trained model to fit a cross-individual P300 detection fine-tuning model that is more suitable for specific new users;

[0014] The online prediction module uses the cross-individual P300 detection pre-trained model or the cross-individual P300 detection fine-tuning model to perform prediction analysis on the samples collected during the online manipulation process and processed by the preprocessing and feature extraction modules, and outputs the results.

[0015] Furthermore, before using the system for cross - individual brain - computer interface control, it is necessary to complete a cross - individual P300 detection pre - training model establishment process in one go beforehand; this process only needs to be carried out once during system development for establishing the cross - individual P300 detection pre - training model. Subsequently, all users will use this model as the cross - individual P300 detection pre - training model without the need to perform this process multiple times. In this process, a large number of subjects need to be recruited for data collection: the subjects need to complete the specified tasks according to the prompts of the stimulus presentation module. At the same time, the signal acquisition module acquires the EEG signals of the subjects during the tasks. After completing the large - scale data collection, the EEG signals of all different subjects during the entire data collection process form a multi - individual large - scale dataset. The signal pre - processing and feature extraction module pre - processes and extracts features from the EEG signals. Finally, the classifier establishment and pre - training module establishes a machine - learning model and uses the EEG sample features and labels of the multi - individual large - scale dataset to train the cross - individual P300 detection pre - training model; thus, the cross - individual P300 detection pre - training model establishment process ends. In the user control and use stage, the stimulus presentation module presents stimuli to the user, and the signal acquisition module acquires the user's EEG signals. The acquired EEG signals are pre - processed and feature - extracted in real - time through the signal pre - processing and feature extraction module, and then passed through the online prediction module. The cross - individual P300 detection pre - training model is used to make predictions and output the results. If, for a certain user, the effect obtained using the cross - individual P300 pre - training model is not good, or there are higher requirements for performance, then let this user perform a short calibration data collection, and based on the cross - individual P300 detection pre - training model, the classifier fine - tuning module uses the newly collected calibration data to fine - tune the classifier to obtain a cross - individual P300 detection fine - tuning model, and in the online prediction module, this model is used to replace the cross - individual P300 detection pre - training model.

[0016] Furthermore, the stimulus presentation module is presented through a computer monitor. The graphical user interface of the stimulus presentation module is a matrix virtual button interface of an m×n arranged in squares, where m and n are positive integers; each square corresponds to a character and can also correspond to various control commands required to control external devices. The working mode of the stimulus presentation module is that the virtual buttons flash in a random order in turns with time overlap, specifically: when the button is not flashing, it is white characters on a black background, and when flashing, it becomes black characters on a green background. After t1 ms from the start point of the flashing of one button, the next button starts to flash. Each flashing duration is the preset t2 ms, where t1 < t2, which means that there is a (t2–t1)ms overlap in the flashing time between two adjacent flashing buttons in time. Each round of flashing means that all m×n characters flash once in a random order; each trial corresponds to a character output or a command sending and contains R rounds of flashing, where R is a positive integer.

[0017] Furthermore, the signal acquisition module includes an electrode cap and an electroencephalogram. The electrode cap contains 32 electrodes, and the contact impedance between the electrodes and the user's scalp is reduced by injecting conductive glue; the electroencephalogram samples, amplifies and denoises the scalp EEG signals at 250Hz, and transmits the processed signals to a computer connected to the electroencephalogram.

[0018] Furthermore, the signal preprocessing and feature extraction module first uses a 0.5-10 Hz bandpass filter to filter the signal of each electrode channel collected by the signal acquisition module, and then extracts the multi-channel EEG signal corresponding to 0-600 ms when each virtual key flashes as the EEG signal corresponding to the key, and subtracts the baseline from the signal, and finally downsamples it with a sampling rate of 6; the baseline calculation method is: take the average value of the multi-channel EEG signal corresponding to 0-200 ms before each virtual key flashes, and use the average value of each channel as the baseline.

[0019] Furthermore, the classifier establishment and pre-training module is only used by developers to perform one-time cross-individual model establishment and model pre-training during the development stage, and the model is used for all new users; first, a cross-individual P300 detection pre-training model is established, and then the cross-individual P300 detection pre-training model is trained with a large number of subject data collected in the cross-individual P300 detection pre-training model establishment link that has been processed by the signal preprocessing and feature extraction modules.

[0020] Furthermore, the classifier fine-tuning module is only used for a certain user when the effect obtained by using the cross-individual P300 detection pre-training model is not good, or when there are higher requirements for performance; in this case, the user collects a short period of calibration data in the same way as the method of collecting samples for a large number of subjects in the establishment of the cross-individual P300 detection pre-training model, but the required amount of calibration data is small. The calibration data is first processed by the signal preprocessing and feature extraction modules, and the cross-individual P300 detection pre-training model is used as a basis to fine-tune the model using the calibration data. The fine-tuned model, namely the cross-individual P300 detection fine-tuning model, is used to replace the cross-individual P300 detection pre-training model in the online prediction module to judge the user's brain signals.

[0021] Furthermore, during online operation, the online prediction module inputs the signal collected by the signal acquisition module and processed by the signal preprocessing and feature extraction modules into the cross-individual P300 detection pre-training model or the cross-individual P300 detection fine-tuning model in real time to obtain a score value for each virtual key being a target key. After R rounds of flashing, the scores of all m×n buttons are compared, and the button with the highest score is used as the prediction target and displayed on the computer monitor.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. The present invention greatly shortens or even completely eliminates the calibration process for new users and new EEG devices, and achieves an accuracy rate close to that of traditional brain-computer interface systems that require calibration for subjects. 150 subjects participated in the calibration data collection in the preparation stage, and 50 users participated in the test. The experimental results show that on the experimental system with 40 virtual buttons, all users achieved significant experimental results, with an average accuracy of 83% when the classifier fine-tuning module was not used and an average accuracy of 91% when the classifier fine-tuning module was used. This result verifies the effectiveness of the present invention, proving that the present invention can broaden the application scenarios of brain-computer interfaces and greatly improve user experience.

[0024] 2. The present invention uses EEG signals, which has the advantages of simple and non-invasive collection compared with other brain signal collection methods.

[0025] 3. The convolutional neural network structure adopted in the present invention is relatively simple, which reduces the amount of calculation during model training and prediction, and is easy to port to other application platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the system of the present invention.

[0027] Figure 2 Schematic diagram of the graphical user interface for the 4×10 matrix used to induce P300.

[0028] Figure 3 Figure 3 is a diagram of the convolutional neural network structure used for cross-individual P300 detection.

[0029] Figure 4 This is the electrode distribution diagram of the electrode cap. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0031] This embodiment provides a cross-individual brain-computer interface system based on convolutional neural network and EEG big data, such as Figure 1 As shown, it specifically includes the following functional modules:

[0032] A stimulus presentation module presents visual stimuli through a graphical user interface to induce the user's P300 potential;

[0033] A signal acquisition module, used to collect the user's EEG signal;

[0034] A signal preprocessing and feature extraction module is used to preprocess the EEG signal, including removing noise and baseline, and extracting features from the preprocessed signal;

[0035] The classifier building and pre-training module is used to build a convolutional neural network model, namely a CNN model, and optimize the model parameters using a large-scale EEG data set collected from a large number of subjects, and fit a cross-individual P300 detection pre-training model based on a convolutional neural network;

[0036] The classifier fine-tuning module uses a small number of new user samples to fine-tune the cross-individual P300 detection pre-trained model to fit a cross-individual P300 detection fine-tuning model that is more suitable for specific new users;

[0037] The online prediction module uses the cross-individual P300 detection pre-trained model or the cross-individual P300 detection fine-tuning model to perform prediction analysis on the samples collected during the online manipulation process and processed by the preprocessing and feature extraction modules, and outputs the results.

[0038] The stimulus presentation module program is written in C++ language combined with MFC library. The graphical user interface presented when the program is running is displayed on a monitor connected to the computer to give the user task prompts and present flickering stimuli for inducing P300 potentials. The signal acquisition module transmits the signal to the computer connected to the electroencephalogram through a USB cable. The signal preprocessing and feature extraction module completes signal filtering, signal slicing, baseline removal, downsampling, and feature extraction. In the development stage, the developer needs to perform the above processing on the EEG signals collected from a large number of subjects, and then use the classifier establishment and pretraining module to establish and pretrain the CNN model for cross-individual P300 detection (i.e., cross-individual P300 detection pretraining model) at one time. In the user use stage, while collecting the user's EEG signal, the EEG signal is preprocessed and feature extracted as above, and then the online prediction module is used to predict the user's intention and output the result. If the pre-trained model does not produce good results for a certain user, or if there is a higher requirement for performance, the user is asked to collect a short period of calibration data, and based on the cross-individual P300 detection pre-trained model, the classifier is fine-tuned using the newly collected calibration data through the classifier fine-tuning module to obtain a cross-individual P300 detection fine-tuning model. This model is used in the online prediction module to replace the cross-individual P300 detection pre-trained model.

[0039] For the paradigm of induced P300 potential in the stimulus presentation module, this embodiment uses a 4×10 matrix graphical user interface, such as Figure 2 As shown. Each time a command is issued (corresponding to a trial), all 40 buttons flash continuously in a random order. When the button is not flashing, it is white text on a black background. When it flashes, it becomes black text on a green background. Each flash lasts for 100ms, and the interval between two adjacent flashes is 30ms. One round includes 40 button flashes (one button corresponds to one flash in each round), and each trial contains 10 rounds of flashes.

[0040] The signal acquisition module includes an electrode cap and an electroencephalogram. The electrode cap contains 32 electrodes, 30 of which are used for EEG signal acquisition. The EEG electrodes are distributed as follows: Figure 4 As shown, there are two other electrodes, one for ground and the other for reference. Conductive glue is injected between the electrode and the user's scalp to reduce contact impedance; the EEG instrument samples, amplifies and denoises the scalp EEG signal, and then transmits the processed signal to the computer.

[0041] The signal preprocessing and feature extraction module first uses a 0.5-10Hz bandpass filter to filter the signal of each electrode channel collected by the signal acquisition module, and then extracts the multi-channel EEG signal corresponding to 0-600ms when each virtual key flashes as the EEG signal corresponding to the key, subtracts the baseline from the signal, and finally downsamples it with a sampling rate of 6.

[0042] The classifier building and pre-training module is only used for one-time cross-individual P300 detection pre-training model building and model pre-training. The convolutional neural network (CNN) model established in this module, such as Figure 3 As shown in the figure. The first layer of CNN is the convolution layer C1, which contains 10 convolution kernels of size C×1, where C is the number of channels. For 30-channel signals, C is 30. The output of the C1 layer is 10 1×25 feature maps. The C1 layer actually completes the linear combination of multiple channels and plays the role of spatial filtering. The second and third layers are also convolution layers, each containing 10 convolution kernels of size 1×8 and 1×10, respectively, which play the role of time domain filtering. The end of CNN is the fully connected layer FC4 and FC5, which contain 15 and 1 neurons respectively. The fully connected layer integrates the local and global information of the previous layers of the network and realizes P300 detection. Among them, all layers except FC5 use rectified linear unit (ReLU) as the activation function; the FC5 layer finally passes through the sigmoid function, and the output value is used as the probability that the sample contains P300 potential. In the development stage, a CNN model for cross-individual P300 detection is obtained by pre-training with data from a large number of subjects that have been processed by the signal preprocessing and feature extraction modules. This can also be called a CNN-based cross-individual P300 detection pre-training model. Traditional brain-computer interfaces require users to complete a certain amount of calibration tasks (generally 10-20 minutes or even longer) according to instructions before use, and collect the user's EEG signals to complete model training for the user. The method proposed in the present invention uses data from a large number of subjects to train the CNN model, which means that new users do not need to perform any calibration tasks before using the brain-computer interface, or complete a small amount (less than 1.5 minutes) of calibration tasks when pursuing high performance.

[0043] The online prediction module works when the user is operating online. When the user is operating the brain-computer interface online, the user's EEG signal is synchronously collected by the signal acquisition module and preprocessed and extracted by the signal preprocessing and feature extraction module, so each signal segment corresponds to a feature matrix. Finally, the matrix is ​​used as the input of the online prediction module to obtain the probability of each key option being the target, and the one with the highest probability is identified as the target, and finally displayed on the graphical user interface.

[0044] The classifier fine-tuning module is only used for users who do not perform well with the pre-trained model or in situations where higher performance is required. Use the stimulus presentation module to present visual stimuli to the user, and use the signal acquisition module and the signal preprocessing and feature extraction module to collect the user's EEG signal and complete preprocessing and feature extraction to obtain the EEG signal characteristics of the new user. Then use the classifier fine-tuning module to fine-tune the cross-individual P300 detection pre-training model of the classifier establishment and pre-training module to fit a detection model that is more suitable for specific new users, that is, the cross-individual P300 detection fine-tuning model. For the above-mentioned cross-individual brain-computer interface system based on convolutional neural networks and EEG big data, the specific process of the experiment is as follows:

[0045] 1) Connect the device. Specifically, the user is given a 30-conductor EEG electrode cap that uses the internationally extended 10-20 standard, and the electrode cap is connected to NuAmps or SynAmps from Neuroscan. 2 Model electroencephalogram, the electroencephalogram is connected to the computer via a USB data cable and a parallel port cable.

[0046] 2) Inject conductive gel and ensure that the contact impedance between the scalp and the electrode meets the acquisition requirements. Specifically, open the Scan 4.5 software of Neuroscan, select the "Acquire" button, and enter the Acquire software. Click the "Z" button in the software to display the contact impedance. Then inject conductive gel on the A2 electrode of the electrode cap, and fix the electrode to the mastoid of the right ear with medical tape, and then inject conductive gel into the GND electrode on the electrode cap. Subsequently, inject conductive gel into each of the 30 EEG electrodes. The contact impedance between the scalp and each electrode is displayed in real time on the computer monitor. This system needs to ensure that the impedance of all 30 electrodes is less than 5kΩ. After completion, click the "■" button to close the impedance display interface.

[0047] 3) The developer builds and trains the model once. Different from the traditional brain-computer interface that is calibrated for each user, this system needs to recruit a large number of subjects (such as 150) for data collection in this link, which is used to train a cross-individual P300 detection pre-training model that can adapt to different users, thereby realizing a cross-individual brain-computer interface, that is, the system does not need to be recalibrated when facing a new user. For each subject, the specific steps of data collection are as follows:

[0048] 3.1) Open SCUTBCI.exe in the computer as an administrator to display the graphical user interface. Click "Parameter Settings (P)" → "Parameter Settings (S)" in the menu. In the pop-up parameter setting window, set "Mode" to "Train" and fill in "Max Trial" with 30, indicating that 30 trials of training data will be collected. Click "OK" to complete the parameter setting.

[0049] 3.2) Click the "S" button in the Scan 4.5 software, and click "Start" in the pop-up window to start the EEG service.

[0050] 3.3) Click "Control (C)" → "Run (R)" in the user graphical interface, and a prompt will pop up saying "Please start the Scan server, then press OK".

[0051] 3.4) Click the "Run" button in Scan 4.5 software to start reading EEG signals. Click the "Send Pulse" button in the "StimGenerator.exe" pop-up in the graphical user interface to test the recording event marker. Then click the "●" button, enter the file name and click "Save" to start saving data.

[0052] 3.5) Click "OK" in the graphical user interface, and the stimulation interface starts flashing. At this time, in each trial, a random character will appear at the top of the graphical user interface, and at the same time, the program will send the event mark corresponding to the target key to the electroencephalogram. Event marks 41-80 represent the 1st to 40th targets on the graphical user interface respectively. The subjects were asked to concentrate on the corresponding characters in the matrix keyboard. Each trial includes 10 rounds of key flashing, and each round of key flashing includes 1 flash for each of the 40 keys. Every time a key flashes, the program will send the event mark corresponding to the flashing key to the electroencephalogram. Event marks 1-40 correspond to the flashing of 40 keys on the graphical user interface.

[0053] 3.6) After each subject completed 30 trials, the developer clicked the “■” button in the Scan 4.5 software to end data collection and recording, and clicked the “S” button to close the EEG instrument service.

[0054] 3.7) The signal preprocessing and feature extraction module first uses a 0.5-10Hz bandpass filter to filter the signal of each electrode channel collected by the signal acquisition module, and then extracts the multi-channel EEG signal corresponding to 0-600ms at the beginning of each virtual key flash as the EEG signal corresponding to the key, and subtracts the baseline from the signal, and finally downsamples it with a sampling rate of 6. In each trial, the preprocessed signals corresponding to multiple flashes of the same key are all superimposed and averaged to obtain the feature matrix of the EEG signals corresponding to 40 keys (one of which corresponds to the target key, contains P300 potential, and is labeled "1"; the other 39 correspond to non-target keys, do not contain P300 potential, and are labeled "0").

[0055] 3.8) After completing the EEG data collection of all subjects, a multi-body large-scale EEG dataset based on P300 brain-computer interface is obtained. A CNN model for cross-individual P300 detection is trained using the feature matrix and labels of all trials of all subjects in the dataset, that is, a cross-individual P300 detection pre-training model based on CNN.

[0056] 4) User online control, the specific steps are as follows:

[0057] 4.1) Open SCUTBCI.exe in the computer as an administrator to display the graphical user interface. Click "Parameter Setting (P)" → "Parameter Setting (S)" in the menu, and set "Mode" to "Online" in the pop-up parameter setting window, indicating online control mode. Click "OK" to complete the parameter setting.

[0058] 4.2) Click the "S" button in the Scan 4.5 software, and click "Start" in the pop-up window to start the EEG service.

[0059] 4.3) Click "Control (C)" → "Run (R)" in the user graphical interface, and a prompt will pop up saying "Please start the Scan server, then press OK".

[0060] 4.4) Click the "Run" button in Scan 4.5 software to start reading EEG signals. Click the "Send Pulse" button in the "StimGenerator.exe" pop-up of the graphical user interface to test the recording event markers.

[0061] 4.5) Click "OK" in the graphical user interface, and the stimulation interface starts flashing. The user focuses on the characters to be spelled on the matrix keyboard. Each trial includes multiple rounds of key flashing, and the number of rounds is determined by the program's confidence in the output results; each round of key flashing includes 1 flash for each of the 40 keys. At the same time as each key flashes, the program will send the event mark corresponding to the flashing key to the EEG instrument, marked 1-40, corresponding to the 40 keys on the graphical user interface.

[0062] 4.6) The signal preprocessing and feature extraction module first uses a 0.5-10Hz bandpass filter to filter the signal of each electrode channel collected by the signal acquisition module, and then extracts the multi-channel EEG signal corresponding to 0-600ms at the beginning of each virtual key flash as the EEG signal corresponding to the key, and subtracts the baseline from the signal, and finally downsamples it with a sampling rate of 6. In each trial, we superimpose and average all the preprocessed signals corresponding to each flash of the same key to obtain the feature matrix of the EEG signals corresponding to 40 keys.

[0063] 4.7) The online prediction module uses the feature matrix corresponding to the 40 keys as the input of the cross-individual P300 detection pre-training model. The model output can be regarded as the probability of each key being the target. The key with the largest probability is identified as the target and finally displayed on the graphical user interface.

[0064] 5) Calculate the accuracy of the user's use of the system. If the accuracy is low, or a higher accuracy is required, perform user-specific calibration. The specific steps are as follows:

[0065] 5.1) Collect data for the new user, as in steps 3.1)-3.7), but the calibration of 30 trials in step 3.6) can be shortened to 5 trials, which takes less than 1.5 minutes.

[0066] 5.2) After completing the user's EEG data collection, a small EEG dataset based on the P300 brain-computer interface is obtained. Using the sample feature matrix and labels of the dataset, the model is fine-tuned based on the cross-individual P300 detection pre-trained model to obtain a fine-tuned model: a cross-individual P300 detection fine-tuning model based on CNN.

[0067] 5.3) User online control is the same as step 4), but the cross-individual P300 detection fine-tuning model obtained in step 5.2) needs to be used instead of the cross-individual P300 detection pre-training model in step 4.7).

[0068] In summary, the present invention designs a cross-individual brain-computer interface system based on convolutional neural networks and EEG big data. The system can establish a cross-individual EEG classification model by learning the commonality of EEG features of different individuals from multiple large-scale data sets, thereby realizing a zero-calibration P300 brain-computer interface. The system can also fine-tune the classification model after collecting a small amount of calibration data from the user, so as to achieve the effect of greatly shortening the calibration time while ensuring a high accuracy rate. Compared with the existing traditional P300 brain-computer interface of the same paradigm, the system greatly shortens or even completely eliminates the cumbersome calibration process for users, and can be cross-used between different models of electroencephalograms, achieving a high accuracy rate, and realizing a high-performance brain-computer interface with less calibration or even zero calibration, thereby greatly simplifying the use of brain-computer interfaces and broadening the use scenarios of brain-computer interfaces. In short, the system of the present invention also provides an effective way to explore the role of EEG big data in high-performance brain-computer interfaces, has potential social value, and is worth promoting.

[0069] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cross-individual brain-computer interface system based on convolutional neural networks and EEG big data, characterized by: include: A stimulus presentation module presents visual stimuli through a graphical user interface to induce the user's P300 potential; A signal acquisition module, used to collect the user's EEG signal; A signal preprocessing and feature extraction module is used to preprocess the EEG signal, including removing noise and baseline, and extracting features from the preprocessed signal; The classifier building and pre-training module is used to build a convolutional neural network model, namely a CNN model, and optimize the model parameters using a large-scale EEG data set collected from a large number of subjects, and fit a cross-individual P300 detection pre-training model based on a convolutional neural network; The classifier fine-tuning module uses a small number of new user samples to fine-tune the cross-individual P300 detection pre-trained model to fit a cross-individual P300 detection fine-tuning model that is more suitable for specific new users; The online prediction module uses the cross-individual P300 detection pre-trained model or the cross-individual P300 detection fine-tuned model to perform prediction analysis on the samples collected during the online manipulation process and processed by the pre-processing and feature extraction modules, and outputs the results; Before using the system for cross-individual brain-computer interface control, it is necessary to complete a cross-individual P300 detection pre-training model establishment link in advance; this link only needs to be performed once during system development to establish a cross-individual P300 detection pre-training model, and subsequent users will use this model as the cross-individual P300 detection pre-training model, and there is no need to perform this link multiple times; in this link, a large number of subjects need to be recruited to participate in data collection: the subjects need to complete the specified tasks according to the prompts of the stimulus presentation module, and at the same time, the signal acquisition module collects the EEG signals of the subjects when performing the tasks; after completing the large-scale data collection, the EEG signals of all different subjects during the data collection process form a multi-body large-scale data set; the EEG signals are pre-processed and feature extracted through the signal preprocessing and feature extraction modules; finally, the classifier establishment and pre-training module establishes a machine learning model, and uses the EEG sample features and labels of the multi-body large-scale data sets to train the cross-individual P300 detection pre-training model; at this point, the cross-individual P300 detection pre-training model establishment link is completed; During the user operation phase, the stimulus presentation module presents stimuli to the user, and the signal acquisition module collects the user's EEG signal. The collected EEG signal is preprocessed and feature extracted in real time through the signal preprocessing and feature extraction module, and then passes through the online prediction module, using the cross-individual P300 detection pre-training model to make predictions and output the results; If the effect of using the cross-individual P300 pre-training model is not good for a certain user, or there are higher requirements for performance, the user is asked to collect a short period of calibration data, and based on the cross-individual P300 detection pre-training model, the classifier is fine-tuned using the newly collected calibration data through the classifier fine-tuning module to obtain the cross-individual P300 detection fine-tuning model, and the online prediction module uses this model instead of the cross-individual P300 detection pre-training model.

2. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 1, characterized in that: The stimulus presentation module is presented through a computer display, and the graphical user interface of the stimulus presentation module is a block arrangement. m × n Matrix virtual key interface, where m , n is a positive integer; each square corresponds to a character, and can also correspond to various control commands required to control external devices; the working mode of the stimulus presentation module is that the virtual keys flash in random order in turn and overlap in time, specifically: when the button is not flashing, it is white text on a black background, and when it flashes, it becomes black text on a green background. After a key flashes starting point, t 1 ms, the next key starts flashing, and each flash lasts for the preset duration. t 2ms, where t 1< t 2, which means that the two buttons flashing adjacently have a certain flashing time ( t 2 – t 1) ms overlap, each round of flashing refers to all m × n Each character flashes once in a random order; each trial corresponds to a character output or a command sent, including R The wheel flashes, R Is a positive integer.

3. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 1, characterized in that: The signal acquisition module includes an electrode cap and an electroencephalogram. The electrode cap contains 32 electrodes. Conductive glue is injected between the electrodes and the user's scalp to reduce contact impedance. The electroencephalogram performs 250 Hz sampling, amplification and denoising on scalp EEG signals, and transmits the processed signals to a computer connected to the electroencephalogram.

4. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 2, characterized in that: The signal preprocessing and feature extraction module first uses a 0.5-10 Hz bandpass filter to filter the signal of each electrode channel collected by the signal acquisition module, and then extracts the multi-channel EEG signal corresponding to 0-600 ms when each virtual key flashes as the EEG signal corresponding to the key, and subtracts the baseline from the signal, and finally downsamples it with a sampling rate of 6; the baseline calculation method is: take the average value of the multi-channel EEG signal corresponding to 0-200 ms before each virtual key flashes, and the average value of each channel is used as the baseline.

5. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 1, characterized in that: The classifier establishment and pre-training module is only used by developers to perform one-time cross-individual model establishment and model pre-training during the development stage, and the model is used for all new users; first, a cross-individual P300 detection pre-training model is established, and then the cross-individual P300 detection pre-training model is trained with a large number of subject data collected in the cross-individual P300 detection pre-training model establishment link that has been processed by the signal preprocessing and feature extraction modules.

6. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 1, characterized in that: The classifier fine-tuning module is only used for a certain user when the effect obtained by using the cross-individual P300 detection pre-training model is not good, or when there are higher requirements for performance; in this case, the user collects a short period of calibration data in the same way as the method of collecting samples for a large number of subjects in the establishment of the cross-individual P300 detection pre-training model, but the required amount of calibration data is small. The calibration data is first processed by the signal preprocessing and feature extraction modules, and the cross-individual P300 detection pre-training model is used as a basis to fine-tune the model using the calibration data. The fine-tuned model, namely the cross-individual P300 detection fine-tuning model, is used to replace the cross-individual P300 detection pre-training model in the online prediction module to judge the user's brain signals.

7. The cross-individual brain-computer interface system based on convolutional neural network and EEG big data according to claim 2, characterized in that: During online control, the online prediction module inputs the signal collected by the signal acquisition module and processed by the signal preprocessing and feature extraction modules into the cross-individual P300 detection pre-training model or the cross-individual P300 detection fine-tuning model in real time, and obtains the score value of each virtual key being the target key. R After the wheel flashes, compare all m × n The button with the highest score is selected as the predicted target and displayed on the computer monitor.

Images