An online adaptive classification method for electroencephalogram signals based on self-supervised learning

Abstract

The present invention discloses an online adaptive classification method for electroencephalogram signals based on self-supervised learning, which includes: obtaining a sample to be measured, which is obtained by preprocessing the electroencephalogram signals collected from the subject to be measured under RSVP; constructing datasets corresponding to the sample to be measured for two self-supervised tasks respectively, to obtain a temporal verification task dataset and a masked space recognition task dataset; updating the parameters of a pre-trained original electroencephalogram signal classification network based on the datasets of the two self-supervised tasks to obtain an updated electroencephalogram signal classification network; using the updated electroencephalogram signal classification network to classify the sample to be measured to obtain a classification result. The original electroencephalogram signal classification network is obtained by training a preset network using a sample dataset, and the sample dataset is obtained based on the electroencephalogram signals obtained from multiple sample subjects in a rapid serial visual presentation experiment. The present invention can solve the problem of distribution shift and improve the classification performance of electroencephalogram signals in the RSVP paradigm.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing, and particularly relates to an online adaptive classification method for electroencephalogram signals based on self-supervised learning. Background Art

[0002] A brain-computer interface (BCI) system establishes a non-invasive connection between the human brain and external devices to achieve information exchange. The BCI system based on electroencephalogram (EEG) signals was first developed to help patients communicate, move, and rehabilitate by converting mental intentions into control commands. As one of the technical means for studying the brain, EEG signals are also often used in neuroscience research. Researchers use EEG signals to unbiasedly measure an individual's fatigue and emotional levels and develop applications such as cognitive and emotion detection. In recent years, more and more EEG-based BCI systems aim to enhance the working ability and efficiency of healthy users by enabling human-computer interaction. In current BCI systems, a user's brain activity is usually monitored using EEG signals, which are recorded from the user's scalp.

[0003] Currently, Rapid Serial Visual Presentation (RSVP), also known as fast serial visual presentation, as a human augmentation experimental paradigm, has received extensive attention from researchers. This experimental paradigm is most commonly used in fields such as counterintelligence, security, and medicine, because in these fields, professionals need to review a large number of images or information. By using the RSVP paradigm and an EEG-based BCI system for EEG signal classification, objects and related information fragments can be detected and identified faster than manual analysis, which greatly improves the work efficiency of professionals. Generally, the RSVP paradigm displays images in sequence at a frequency of 5–20 Hz, where the ratio of non-target images to target images is approximately 10:1. This helps induce event-related potential (ERP) components in EEG signals, which are related to the brain's attention mechanism and memory processing. In recent years, researchers have been working on improving the classification performance of EEG signals in the RSVP paradigm.

[0004] In existing electroencephalogram (EEG) signal classification methods, traditional machine learning-based methods usually adopt handcrafted features, such as statistical features in the time domain, band power in the frequency domain, and discrete wavelet transform features in the time-frequency domain. Subsequently, these features are fed into linear discriminant analysis (LDA) or Fisher linear discriminant analysis (FLD) algorithms for classification. For example, Blankertz et al. proposed a regularized linear discriminant analysis algorithm (rLDA) to accurately estimate the covariance matrix in a high-dimensional space. This method uses a shrinkage estimator to form a regularized version of linear discriminant analysis, and its performance is superior to other linear discriminant analysis-based methods. Parra et al. proposed a hierarchical discriminant component analysis algorithm (HDCA), which first uses FLD to train spatial weights, and then trains a logistic regression classifier to learn temporal weights and achieve classification. Xiao et al. developed an algorithm called standard pattern matching. This algorithm constructs discriminative spatial patterns and canonical correlation analysis patterns, and then matches these two patterns to form a robust classifier.

[0005] It is worth noting that the recently developed deep learning methods are taking a dominant position in the field of EEG signal classification. Different from traditional research that relies on expert-level experience and prior domain knowledge to extract feature information, deep learning can automatically extract discriminative features from brain activities. For example, Schirrmeister et al. first proposed an end-to-end deep network called DeepConvNet for EEG decoding tasks. This model decodes task-related information from raw EEG without handcrafting features, highlighting the potential of combining deep convolutional neural networks with advanced visualization techniques for brain mapping. Lawhern et al. proposed a compact neural network called EEGNet and achieved good performance in various EEG classification tasks. It uses depthwise convolution and separable convolution to construct an EEG-specific network, which encapsulates multiple EEG feature extraction concepts, such as optimal spatial filtering and filter bank construction. Vázquez et al. proposed EEG-Inception, which first integrates Inception modules to efficiently extract temporal features at different time scales for ERP classification. Considering the phase-locking characteristics of event-related potential (ERP) components, Zang et al. proposed PLNet to learn phase information to improve the classification performance of EEG signals.

[0006] However, EEG signals are often non-stationary in reality. Between two experiments, their data distributions tend to change over time. The non-stationarity of EEG signals can be caused by various events, such as changes in the user's attention level, electrode placement, or user fatigue. Neuroscience research shows that the root cause of EEG signal non-stationarity is not only related to the impact of external stimuli on brain mechanisms but also to the conversion of the inherent metastability related to cognitive tasks of neural components. Thus, in practical EEG applications, non-stationary EEG signals can lead to changes in the distribution between training data and test data over time, which greatly limits the performance of EEG classification under the RSVP paradigm.

[0007] However, most existing EEG signal classification methods do not consider the distribution shift problem and assume that the distribution of EEG data does not change between training data and test data. For the distribution shift problem, adversarial robustness and domain adaptation are among the few solutions. They attempt to predict the difference between the training and test distributions through topological structures or test distribution data. However, since test data is not introduced, it is difficult for adversarial robustness to accurately predict the test distribution. For domain adaptation, due to more and more privacy issues, ever-expanding datasets, and many other real-world limitations, it may be impractical to revisit the training data during testing.

[0008] Therefore, how to solve the distribution shift problem and improve the classification performance of EEG signals in the rapid serial visual presentation paradigm is an urgent problem to be solved in this field. Summary of the Invention

[0009] The purpose of the embodiments of the present invention is to provide an online adaptive classification method for EEG signals based on self-supervised learning to achieve the purpose of solving the distribution shift problem and improving the classification performance of EEG signals in the rapid serial visual presentation paradigm. The specific technical solutions are as follows:

[0010] An online adaptive classification method for EEG signals based on self-supervised learning includes:

[0011] Obtaining a sample to be measured; wherein, the sample to be measured is obtained by preprocessing the electroencephalogram signals collected from the subject to be measured under rapid serial visual presentation.

[0012] Constructing datasets corresponding to the sample to be measured for two self-supervised tasks respectively to obtain a temporal verification task dataset and a masked space recognition task dataset.

[0013] Based on the time series verification task dataset and the masked space recognition task dataset, update the parameters of the pre-trained original electroencephalogram (EEG) signal classification network to obtain an updated EEG signal classification network; wherein, the original EEG signal classification network is obtained by training a preset network using a sample dataset, and the sample dataset is obtained from the electroencephalogram signals of multiple sample subjects in a rapid serial visual presentation experiment.

[0014] Use the updated EEG signal classification network to classify the sample to be tested, and obtain the corresponding classification result.

[0015] In an embodiment of the present invention, the construction of the datasets corresponding to the two self-supervised tasks for the sample to be tested, to obtain the time series verification task dataset and the masked space recognition task dataset, includes:

[0016] For the time series verification task, use the sample to be tested as the positive sample, and obtain negative samples by performing swapping operations on different parts of the sample to be tested in the time dimension. The negative samples with corresponding labels and the positive sample together constitute the time series verification task dataset;

[0017] For the masked space recognition task, according to the preset correspondence between brain regions and electrodes, divide the sample to be tested into a preset number of regions, and each time use a preset noise to mask the electroencephalogram signal of a non-repeated region in the sample to be tested, to obtain the masked sample to be tested with a label for this time; The masked samples to be tested obtained each time together constitute the masked space recognition task dataset; wherein, the label of the masked sample to be tested obtained each time is the same as the region number of the electroencephalogram signal region of the sample to be tested masked this time.

[0018] In an embodiment of the present invention, the obtaining of negative samples by performing swapping operations on different parts of the sample to be tested in the time dimension includes:

[0019] Divide the sample to be tested into two parts, front and back, in the time dimension;

[0020] Swap the electroencephalogram signals corresponding to the front and back parts respectively to obtain the negative sample corresponding to the sample to be tested.

[0021] In an embodiment of the present invention, the preset correspondence between brain regions and electrodes includes a preset number of brain regions and the names of multiple electrodes corresponding to each brain region for collecting electroencephalogram signals.

[0022] In an embodiment of the present invention, the preset noise includes Gaussian noise.

[0023] In one embodiment of the present invention, updating the parameters of the pre-trained original electroencephalogram (EEG) signal classification network based on the timing verification task data set and the masked space recognition task data set to obtain an updated EEG signal classification network includes:

[0024] Inputting the timing verification task data set and the masked space recognition task data set into the original EEG signal classification network simultaneously to respectively obtain the features of the timing verification task and the features of the masked space recognition task;

[0025] Inputting the features of the timing verification task into a pre-constructed timing verification task classification head to obtain a predicted classification result of the timing verification task; and inputting the features of the masked space recognition task into a pre-constructed masked space recognition task classification head to obtain a predicted classification result of the masked space recognition task;

[0026] Calculating a network loss according to the labels of the timing verification task data set, the predicted classification result of the timing verification task, the labels of the masked space recognition task data set, and the predicted classification result of the masked space recognition task, and updating the parameters of the original EEG signal classification network by using the network loss to obtain an updated EEG signal classification network.

[0027] In one embodiment of the present invention, the timing verification task classification head and / or the masked space recognition task classification head are composed of two fully connected layers.

[0028] In one embodiment of the present invention, the process of obtaining the sample data set includes:

[0029] Performing a rapid serial visual presentation experiment on the selected sample subjects, and collecting the electroencephalogram signals of each sample subject under preset experimental conditions;

[0030] Preprocessing the collected electroencephalogram signals;

[0031] Constituting a sample data set from all the preprocessed electroencephalogram signals, and dividing the sample data set into a training set and a test set according to a preset ratio.

[0032] In one embodiment of the present invention, for any electroencephalogram signal, the preprocessing process includes:

[0033] Segmenting the collected electroencephalogram signal according to the timestamps of the occurrences of each stimulus in the rapid serial visual presentation corresponding to the electroencephalogram signal to obtain a plurality of data segments; wherein, each occurrence of an image in the image sequence corresponding to the rapid serial visual presentation corresponds to a stimulus;

[0034] Filtering each data segment;

[0035] Perform downsampling processing on each filtered data segment;

[0036] Each data segment after downsampling is normalized.

[0037] In one embodiment of the present invention, the preset network includes an EEGNet network.

[0038] Beneficial effects of the present invention:

[0039] In the online adaptive classification method of EEG signals based on self-supervised learning provided by the embodiment of the present invention, the sample data set obtained by using the EEG signals obtained by the sample subjects in the rapid sequence visual presentation experiment is used to train the preset network to obtain a trained model, that is, the original EEG signal classification network. In the actual test, for each sample to be tested obtained after preprocessing the EEG signals collected by the test subjects under the rapid sequence visual presentation, two self-supervised tasks including the timing verification task and the mask space recognition task are first constructed, and the timing verification task data set and the mask space recognition task data set are obtained; then, based on the timing verification task data set and the mask space recognition task data set, the parameters of the original EEG signal classification network are updated to obtain an updated EEG signal classification network; finally, the updated EEG signal classification network is used to classify the sample to be tested to obtain the corresponding classification result.

[0040] Since the unlabeled test samples presented during the actual test provide information about their data distribution. The method of the embodiment of the present invention converts a single unlabeled test sample into a self-supervised learning problem, including a time series verification task and a mask space recognition task. By utilizing the self-supervised learning of the test samples, the parameters of the model trained using the test data are updated before prediction to fully extract the distribution information of the EEG signals during the actual test, and the parameters of the feature extractor are updated during the actual test, thereby narrowing the distribution difference of the EEG signals during training and actual testing, so that the updated model fully adapts to the data distribution of the samples during the actual test, thereby improving the classification performance of the EEG signals.

[0041] Compared with the solution to the traditional EEG signal distribution offset problem, the method of the embodiment of the present invention does not predict the distribution change of EEG data between training data and test data, but learns from the samples to be tested during the actual test, and solves the distribution offset problem more efficiently by making full use of the data signals of the samples to be tested during the actual test. In addition, the training set data is not used to solve the distribution offset problem during the actual test. The embodiment of the present invention can solve the distribution offset problem and improve the classification performance of EEG signals in the rapid sequence visual presentation paradigm. It can be used for services in multiple fields such as medical care and has high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of a method for online adaptive classification of electroencephalogram signals based on self-supervised learning provided by an embodiment of the present invention;

[0043] Figure 2 Schematic block diagram of the overall implementation process of the method for online adaptive classification of electroencephalogram signals based on self-supervised learning according to an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of a construction process of negative samples when constructing a time series verification task data set according to an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of a process for constructing a masked space recognition task data set according to an embodiment of the present invention;

[0046] FIG. 5(a) is a schematic diagram of a non-target image in an image sequence corresponding to rapid serial visual presentation according to an embodiment of the present invention;

[0047] FIG. 5(b) is a schematic diagram of a target image in an image sequence corresponding to rapid serial visual presentation according to an embodiment of the present invention;

[0048] Figure 6 Task time sequence diagram for collecting electroencephalogram signals in the experiment according to an embodiment of the present invention;

[0049] Figure 7 Schematic diagram of the structure of the EEGNet network. Detailed implementation manners

[0050] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] To solve the problem of distribution shift and improve the classification performance of electroencephalogram signals in the rapid serial visual presentation paradigm, an embodiment of the present invention provides a method for online adaptive classification of electroencephalogram signals based on self-supervised learning.

[0052] It should be noted that the execution subject of a method for online adaptive classification of electroencephalogram signals based on self-supervised learning provided by an embodiment of the present invention can be a device for online adaptive classification of electroencephalogram signals based on self-supervised learning, and this device can run in an electronic device. Among them, the electronic device can be a server or a terminal device, but is not limited thereto.

[0053] Please refer to Figure 1 and Figure 2Understand an online adaptive classification method for electroencephalogram signals based on self-supervised learning provided by an embodiment of the present invention. Figure 1 It is a schematic flowchart of an online adaptive classification method for electroencephalogram signals based on self-supervised learning provided by an embodiment of the present invention; Figure 2 It is a schematic block diagram of the overall implementation process of the online adaptive classification method for electroencephalogram signals based on self-supervised learning in an embodiment of the present invention.

[0054] As Figure 1 shown, an online adaptive classification method for electroencephalogram signals based on self-supervised learning provided by an embodiment of the present invention may include the following steps:

[0055] S1, obtain a sample to be tested.

[0056] For each sample to be tested obtained in an embodiment of the present invention, the provided online adaptive classification method for electroencephalogram signals based on self-supervised learning can be separately executed. In each step, one sample to be tested is taken as an example for illustration.

[0057] Among them, the sample to be tested is obtained after preprocessing the electroencephalogram signal collected from the subject to be tested under rapid serial visual presentation.

[0058] The subject to be tested in an embodiment of the present invention is a person whose health conditions meet the requirements, that is, who can generate ERP components under the rapid serial visual presentation paradigm. The rapid serial visual presentation corresponds to an image sequence containing a target image and a non-target image, and can contain at least one image sequence. Each image sequence appears sequentially on the display screen, and each image sequence contains a target image and a non-target image. Among them, the target image contains a target, and the non-target image does not contain a target. The target can be set as needed, for example, it can be a person, a vehicle, an animal, etc. It can be understood that when the subject observes the image sequence corresponding to the rapid serial visual presentation, ERP components will be generated and reflected in the EEG signal, and the electroencephalogram signal of the subject to be tested under the rapid serial visual presentation can be collected by using an EEG signal acquisition device.

[0059] During the collection process, the subject to be tested wears an electrode cap, and the electroencephalogram signal of the subject to be tested is collected through the electrodes on the electrode cap. According to different test devices, the configuration of relevant parameters in the collection process can be different. For example, in an optional implementation manner, a 64-channel electroencephalogram electrode cap can be used, the sampling rate can be 1024Hz, etc., and electroencephalogram paste is applied to keep the impedance of each electrode below 25kΩ to ensure obtaining high-quality electroencephalogram signals. The process of collecting the electroencephalogram signal of the subject to be tested under the rapid serial visual presentation can be understood in combination with the process of obtaining the sample data set described later.

[0060] In an optional implementation, for any electroencephalogram signal, such as the electroencephalogram signal corresponding to the sample to be measured, the preprocessing process may include:

[0061] A1. According to the timestamps of the occurrences of the stimuli in the rapid serial visual presentation corresponding to the electroencephalogram signal, segment the collected electroencephalogram signal to obtain a plurality of data segments;

[0062] When any subject (for S1, it is the subject to be measured) observes the image sequence corresponding to the rapid serial visual presentation, the appearance of each image in the image sequence corresponding to the rapid serial visual presentation corresponds to a stimulus, and the occurrence time of each image in the image sequence is knowable and can be marked with a timestamp. Therefore, the occurrence time of each stimulus is the time corresponding to the timestamp.

[0063] It can be understood that when the subject observes the image sequence corresponding to the rapid serial visual presentation, only when the target image appears will the subject be stimulated to generate ERP components; moreover, the stimulus given by the image is completed within a short time after the image appears, that is to say, the effective time period for the subject to receive the stimulus and generate ERP components is very short. Therefore, it is only necessary to intercept the electroencephalogram signals within the effective time period of each image for analysis, and excluding the remaining invalid information can reduce the data processing volume of the electroencephalogram signals and improve the processing efficiency.

[0064] Specifically, A1 may include:

[0065] In the electroencephalogram signal, starting from the timestamps of the occurrences of the stimuli in the corresponding rapid serial visual presentation, respectively intercept the electroencephalogram signal data intervals with a preset duration to obtain the data segments corresponding to the stimuli in the electroencephalogram signal.

[0066] Taking the example that the image sequence corresponding to the rapid serial visual presentation corresponding to the electroencephalogram signal contains 50 images, since the appearance of each image is marked with a timestamp, then, in the electroencephalogram signal, starting from each timestamp, respectively intercept a section of the electroencephalogram signal with a preset duration, and a total of 50 data segments can be obtained. Each data segment corresponds to an image, that is, a stimulus.

[0067] Among them, the preset duration is set according to empirical values, such as 1 second, etc., that is, intercept the electroencephalogram data from the start of the appearance of the target or non-target during the rapid serial visual presentation to 1 second after the start.

[0068] A2. Filter each data segment;

[0069] During the process of collecting EEG signals, EEG signals are usually contaminated by noise from different sources. These artifacts may come from blinking, electrocardiogram (ECG), electromyogram (EMG), and any external sources related to the equipment involved in the system. These artifacts may have similar amplitudes as the EEG signal and are likely to interfere with the subsequent tasks. In this step, the purpose of filtering each data segment is to eliminate or attenuate the noise and simplify the subsequent processing operations without losing relevant information so that reliable features can be extracted in the subsequent links.

[0070] This step can be implemented using any existing denoising method, and the present invention does not limit this. For example, in an optional implementation, a bandpass filter can be used, specifically a sixth-order Butterworth bandpass filter with a cutoff frequency of 0.1 to 48 Hz.

[0071] A3, downsampling each data segment after filtering;

[0072] In order to reduce the amount of data for subsequent network feature extraction, this step performs downsampling on each data segment after filtering. For example, the sampling rate of each data segment after filtering can be reduced to 256Hz, etc. Of course, the frequency after the sampling rate is reduced is not limited to this value, and can be reasonably set according to the scene requirements.

[0073] A4, normalize each data segment after downsampling.

[0074] Since the data distribution forms of each data segment after downsampling processing may be different, in order to facilitate subsequent network feature extraction and other processing, each data segment after downsampling processing needs to be normalized to achieve the unification of data distribution form and data format. The specific normalization method can be implemented by selecting any existing normalization method according to the needs. For example, in an optional implementation, the Z scoring method can be used. Please refer to the relevant existing technology for the specific processing process, which will not be described in detail here.

[0075] S2, constructing data sets corresponding to the two self-supervision tasks for the samples to be tested, and obtaining a time series verification task data set and a mask space recognition task data set.

[0076] The embodiment of the present invention constructs two self-supervised tasks for each sample to be tested, namely, a time sequence verification task and a mask space recognition task. The purpose is to learn the temporal contextual connection of the EEG signal by solving the time sequence verification task, and to mine the spatial relationship between the channel areas of the EEG signal by solving the mask space recognition task.

[0077] In an optional implementation, S2 may include:

[0078] S21. For the timing verification task, using the sample to be tested as a positive sample, negative samples are obtained by swapping different parts of the sample to be tested in the time dimension. The negative samples with corresponding labels and the positive sample together constitute a timing verification task data set.

[0079] To solve the timing verification task, the feature extraction network corresponding to the preset network must learn the correlation in the time dimension, which is beneficial to improving the classification accuracy of EEG signals.

[0080] Among them, the label of the positive sample is "positive" and can be represented by 1; the label of the negative sample is "negative" and can be represented by 0. The positive sample and its label, and the negative sample and its label together constitute the timing verification task data set.

[0081] For S21, in an optional implementation, the step of obtaining negative samples by swapping different parts of the sample to be tested in the time dimension may include:

[0082] B1. Divide the sample to be tested into two parts, the front part and the back part, in the time dimension.

[0083] B2. Swap the EEG signals corresponding to the front and back parts respectively to obtain the negative sample corresponding to the sample to be tested.

[0084] Specifically, for the length of the sample to be tested in the time dimension, the sample to be tested can be divided into two parts, the front part and the back part, in the time dimension according to a fixed time position, or it can also be divided into two parts, the front part and the back part, in the time dimension according to a random time position; or, for the length of the sample to be tested in the time dimension, it can also be divided according to a fixed or random ratio, such as 30%. The first 30% part of the length of the sample to be tested in the time dimension is used as the front part, and the latter 70% part is used as the back part, etc. This is all reasonable. Swapping the EEG signals corresponding to the front and back parts respectively will obtain the negative sample.

[0085] Regarding the construction process of the negative sample, please refer to Figure 3 Understand that, among them, the sample to be tested as the positive sample is divided into two parts, the front part and the back part, in the time dimension, which are represented by T1 and T2 respectively. By swapping the EEG signals corresponding to T1 and T2 respectively, the negative sample is obtained. It can be seen that in the negative sample, T2 is in the front and T1 is in the back.

[0086] For S21, in an optional implementation, the step of obtaining negative samples by swapping different parts of the sample to be tested in the time dimension may include:

[0087] C1, divide the sample to be tested into at least three parts in the time dimension;

[0088] C2, exchange the electroencephalogram signals corresponding to the at least three parts respectively to obtain a negative sample corresponding to the sample to be tested.

[0089] Taking the example of dividing the sample to be tested into three parts in the time dimension, for example, the sample to be tested as a positive sample is divided into three parts T1, T2, and T3 in the time dimension. Then, the electroencephalogram signals corresponding to these three parts can be arbitrarily exchanged to obtain a negative sample different from the positive sample. For example, the three consecutive parts of the negative sample in the time dimension can be T2, T1, and T3, or T3, T2, and T1, etc.

[0090] S22. For the mask space recognition task, according to the preset correspondence between brain regions and electrodes, divide the sample to be tested into a preset number of regions, and each time use a preset noise to mask the electroencephalogram signals of a non-repeated region in the sample to be tested to obtain the masked sample to be tested with a label for this time; the masked samples to be tested obtained each time together constitute the mask space recognition task data set;

[0091] Among them, the label of the masked sample to be tested obtained each time is the same as the region number of the region where the electroencephalogram signal of the sample to be tested is masked this time.

[0092] In the embodiment of the present invention, the preset correspondence between brain regions and electrodes includes a preset number of brain regions and the names of multiple electrodes corresponding to each brain region for collecting electroencephalogram signals.

[0093] Those skilled in the art can understand that the brain region can be divided into a preset number of different regions, and some electrodes for collecting EEG signals are corresponding to each region and are placed on the scalp of the subject to be tested or the sample subject in the experiment process. Therefore, for a subject, its EEG signals can be divided into EEG signals of a preset number of regions.

[0094] In an optional implementation manner, since the embodiment of the present invention can collect EEG signals using 64 electroencephalogram channels, the preset number can be 8. Of course, other numbers can also be selected according to needs, which are all reasonable. For the convenience of understanding, the following takes the preset number of 8 as an example for illustration.

[0095] Exemplarily, for the construction process of the mask space recognition task data set, please refer to Table 1 and Figure 4It is understood that Table 1 is a correspondence table between brain regions and electrodes, representing an example of the correspondence between preset brain regions and electrodes, but not limiting the correspondence between preset brain regions and electrodes in the embodiments of the present invention. In Table 1, the brain regions are divided into eight regions, including the forebrain, left temporal lobe, frontal, right temporal lobe, left parietal lobe, occipital lobe, right parietal lobe, and anterior part. Each brain region corresponds to some electrodes, and the names are shown in Table 1.

[0096] Table 1 A correspondence table between brain regions and electrodes

[0097]

[0098]

[0099] And please refer to Figure 4 , Figure 4 , which is a schematic diagram of a process for constructing a masked space recognition task data set in an embodiment of the present invention; the large circle on the left side of the arrow represents the brain, and each small circle represents the corresponding brain region, and the symbol in each small circle represents the electrode name corresponding to the brain region.

[0100] It can be understood that for the EEG signal corresponding to the sample to be tested, the EEG signal can be divided into EEG signals of 8 regions, and each time a non-repeating region of the EEG of the sample to be tested is masked with preset noise, it is a masking operation. For the i-th masking operation, the EEG signal of the i-th region among the EEG signals of these 8 regions is masked with preset noise. Figure 4 In, the EEG signal of the masked region is filled with black and white grids, that is, the Gaussian noise mask in the legend. Then, the new 8-region EEG signal obtained after the i-th masking operation is used as the masked sample to be tested obtained in the i-th time, and its label is i, where i ∈ [1, 8]. According to the above method, 8 masking operations are performed so that the regions masked in each masking operation are not repeated. Then, all the masked samples to be tested corresponding to the 8 times finally obtained together constitute the masked space recognition task data set.

[0101] Figure 4 In, the EEG signals of the 1st to 8th regions corresponding to the EEG signal of the sample to be tested are masked in sequence. Of course, in an alternative embodiment, it is not necessary to follow a certain order, as long as each masking traverses all regions and the regions masked in each masking are not repeated.

[0102] Such as Figure 4 shown, in an alternative embodiment, the preset noise includes Gaussian noise. Of course, the preset noise can also be Poisson noise, multiplicative noise, salt and pepper noise, etc., which can play a masking role.

[0103] S3. Based on the timing verification task dataset and the mask space recognition task dataset, update the parameters of the pre-trained original electroencephalogram (EEG) signal classification network to obtain an updated EEG signal classification network.

[0104] Among them, the original EEG signal classification network is obtained by training a preset network using a sample dataset, and the sample dataset is based on the electroencephalogram signals obtained from multiple sample subjects in a rapid serial visual presentation experiment.

[0105] To facilitate understanding of the solution of the embodiments of the present invention, first, the acquisition process of the sample dataset and the original EEG signal classification network will be described.

[0106] (1) The acquisition process of the sample dataset includes:

[0107] (1) Conduct a rapid serial visual presentation experiment on the selected sample subjects, and collect the electroencephalogram signals of each sample subject under preset experimental conditions;

[0108] In the embodiments of the present invention, rapid serial visual presentation can also be referred to as rapid serial visual presentation, that is, each image in the corresponding image sequence is quickly switched and displayed on the display screen in series.

[0109] In the experimental preparation stage, multiple sample subjects meeting the health requirements are selected. Specifically, all subjects have normal or corrected normal vision, and each sample subject has no neurological problems or serious medical history to avoid affecting the experimental results. The number of sample subjects can be determined according to the sample quantity requirements, such as 10, etc. In addition, the sample subjects need to be informed in detail of the experimental precautions and confirm their consent to carry out the experimental process to ensure that the experimental process meets the relevant requirements.

[0110] The preset experimental conditions can be: the EEG signals are collected using the Active-Two system of Biosemi Company, the sample subjects wear a 64-channel EEG electrode cap, the sampling rate is set to 1024 Hz, and the impedance of each electrode is ensured to be lower than 25 kΩ during the experimental process to ensure the acquisition of high-quality EEG signals, etc.

[0111] In the experimental stage, a rapid serial visual presentation experiment is conducted on each sample subject, and the electroencephalogram signals of the subject are collected through the electrodes on the electrode cap at the same time.

[0112] Each experiment has four states in chronological order, namely the preparatory state, the viewing state, the intermittent state, and the waiting state, where:

[0113] In the preparatory state, first, a preset pattern such as a crosshair will appear on the display screen to help the subject focus and wait to continuously view the picture sequence. After the waiting duration, the picture sequence will be played immediately. The waiting duration can be 2s or the like.

[0114] In the viewing state, an image sequence containing target images and non-target images will appear randomly in the center of the display screen at a predetermined frequency, such as 5Hz or 10Hz, etc., for the sample subjects to view. The number of images in the image sequence can be 50 or the like. Playing one image sequence means completing one viewing and also means completing one trial experiment.

[0115] The styles of the images in the image sequence can refer to Figure 5. Figure 5(a) is a schematic diagram of a non-target image in the image sequence corresponding to the rapid serial visual presentation of the embodiment of the present invention; Figure 5(b) is a schematic diagram of a target image in the image sequence corresponding to the rapid serial visual presentation of the embodiment of the present invention. Figure 5(a) and Figure 5(b) respectively give 2 images as examples. Here, the corresponding images are displayed in grayscale. Among them, the resolution of the images is 800*600 or the like, and the target is a vehicle. Of course, Figure 5(a) and Figure 5(b) are only examples, and the target of the embodiment of the present invention is not limited to vehicles and can be reasonably selected according to needs, such as people, animals, etc.

[0116] In the experiment of the embodiment of the present invention, a total of 500 target images and 1000 non-target images were collected. Each time, 4 target images and 46 non-target images were selected to form 50 images, that is, an image sequence. In the viewing state, after every 50 images are displayed, there will be an intermittent state, and the display screen can be converted to a black screen or other states, and this state can last for 2 seconds or the like to help the sample subjects adjust their states.

[0117] Completing 10 trial experiments according to the above process means completing 1 block experiment. At this time, the sample subjects enter the waiting state to rest. After a predetermined rest duration, such as 4 seconds later, they enter the next block experiment again. This cycle continues until the experiment with a preset number of blocks is completed, and then the experiment process can end. The preset number of blocks can be 30 or the like. It should be added that during the entire experiment process, the images in each image sequence are randomly generated.

[0118] Regarding the timing of electroencephalogram signal acquisition in the experiment of the embodiment of the present invention, please refer to Figure 6 for understanding. Figure 6This is the task timing diagram for collecting electroencephalogram (EEG) signals in the experiments of the embodiments of the present invention. Here, the corresponding images are displayed in grayscale. Among them, there are 30 blocks in the experiment, each block has 10 trials, one image sequence is played in one trial, and one image sequence has 50 images. Through the above experiments, a total of 1200 single-trial samples are collected.

[0119] It can be understood that when obtaining the sample to be measured, the process of collecting the EEG signals of the subject to be measured under rapid serial visual presentation is similar to the above process.

[0120] (2) Preprocess the collected EEG signals;

[0121] In this step, each collected EEG signal used as a sample is preprocessed to obtain the corresponding preprocessed EEG signal. For the preprocessing process, please refer to the understanding of the process for any EEG signal in S1, and no repeated description will be made here.

[0122] (3) Construct a sample data set from all the preprocessed EEG signals, and divide the sample data set into a training set and a test set according to a preset ratio.

[0123] Specifically, a sample data set is constructed from all the preprocessed EEG signals, and the preset ratio for dividing the training set and the test set can be 8:2, etc. The training set is used to train a preset network to obtain an original EEG signal classification network.

[0124] (2) The process of obtaining the original EEG signal classification network includes:

[0125] 1) Build a network structure;

[0126] In the embodiments of the present invention, a preset network is used as the built network structure.

[0127] In an optional implementation manner, the preset network may include an EEGNet network. The use of this network is considered to have the following three advantages: (A) It can be applied to a variety of different BCI paradigms, (B) It can be trained with very limited data, and (C) It can generate neurally physiologically interpretable features.

[0128] For the structure of the EEGNet network, please refer to Figure 7 , and the EEGNet network is composed of a time feature extraction unit, a channel correlation extraction unit, a spatio-temporal feature residual fusion unit, and a classification unit connected in sequence. Specifically:

[0129] The time dynamic extraction unit uses a 1*64 convolution module to extract the time-domain features of the EEG signals by performing convolution operations in the time dimension.

[0130] The channel correlation extraction unit is a spatial feature extraction unit composed of 64*1 convolution modules, which extracts the spatial feature representation of EEG signals by performing depth convolution operations on different EEG channels.

[0131] The spatio-temporal feature fusion unit is formed by cascading a depth convolution operation module and a 1*1 pointwise convolution module, and extracts more robust features by fusing spatio-temporal information.

[0132] The classification unit completes the classification function through a fully connected layer.

[0133] Regarding the specific structures and processing methods of each part of the EEGNet network, please understand them in combination with relevant existing technologies and will not be elaborated here.

[0134] 2) Complete network training

[0135] The network training process of the embodiments of the present invention is carried out with reference to the EEGNet training process. Specifically, the training set is used to perform iterative training on EEGNet by the gradient descent method to obtain a trained model, which is called the original EEG signal classification network; among them, the training parameters can be set as follows: the number of training times is set to 150 times, the single-sample input volume per time is 4, the loss function is the cross-entropy loss function, the optimizer uses the adaptive moment estimation optimizer, and the initial learning rate is 0.001.

[0136] The specific training process may include the following steps:

[0137] ① Each time, 4 single-trial samples are selected from the training set and sent into the EEGNet network. First, the time features of the sample data are extracted, then the channel correlation is extracted, and then the EEG signal features are obtained through the spatio-temporal feature fusion unit, and then the EEG signal features are sent into the convolutional classifier for classification;

[0138] ② Calculate the cross-entropy loss according to the classification result and the true label of the sample, and then the adaptive moment estimation optimizer updates the parameters in the convolutional layer and the batch normalization layer in the EEGNet network according to the cross-entropy loss;

[0139] ③ Traverse all samples in the training set to complete the training and obtain the trained EEGNet network, that is, the original EEG signal classification network.

[0140] In an optional implementation, after using the training set to train the EEGNet network through gradient descent according to the above steps to obtain the original EEG signal classification network, the test set can be used again for testing and parameter optimization. Specifically, the samples in the test set can be directly input into the original EEG signal classification network for classification to obtain classification results, and then the classification results are statistically analyzed. Combining with the true labels of the samples in the test set, the classification accuracy of the original EEG signal classification network on the test set is obtained. According to the classification accuracy, parameters such as the convolutional kernel size and learning rate of the original EEG signal classification network are adjusted, so as to obtain an optimized network with good performance on the offline data set. This optimized network can be used as the trained original EEG signal classification network for real-time fast serial visual presentation EEG signal classification of unknown subjects to be tested. It can be seen that this implementation uses the test set for re-optimization, which can further improve the network performance and the accuracy of EEG signal classification.

[0141] The above is a brief introduction to the acquisition process of the sample data set and the original EEG signal classification network.

[0142] For S3, based on the time series verification task data set and the masked space recognition task data set, the parameters of the pre-trained original EEG signal classification network are updated to obtain the updated EEG signal classification network.

[0143] The embodiment of the present invention uses two self-supervised tasks to update the parameters of the original EEG signal classification network again. Generally speaking, according to the multi-task learning theory, the positive and negative pair data constructed by the two self-supervised tasks are simultaneously input into the original EEG signal classification network. The two self-supervised tasks are helpful for the EEG signal classification task corresponding to the sample to be tested, learn relevant spatial information and temporal information, and promote each other in the process of continuous iterative learning, further improving the model, that is, the classification ability of the original EEG signal classification network.

[0144] In an optional implementation, S3 may include the following steps:

[0145] S31, input the time series verification task data set and the masked space recognition task data set into the original EEG signal classification network at the same time, and respectively obtain the features of the time series verification task and the features of the masked space recognition task;

[0146] Specifically, the EEGNet used in the original EEG signal classification network can be understood as a feature extractor connected in series with a classifier. After inputting the time series verification task dataset and the masked space recognition task dataset into the original EEG signal classification network simultaneously in this step, the feature extractor outputs the features of the time series verification task and the features of the masked space recognition task. These two features are feature vectors with a dimension of 1x128, respectively containing the time and space information of the sample to be tested. Among them, the time information specifically refers to the internal relationship between the components of ERP (event-related potential) in the EEG signal of the sample to be tested, and the space information refers to the relationship between the various EEG channels of the sample to be tested.

[0147] S32. Input the features of the time series verification task into a pre-constructed time series verification task classification head to obtain the predicted classification result of the time series verification task; and input the features of the masked space recognition task into a pre-constructed masked space recognition task classification head to obtain the predicted classification result of the masked space recognition task.

[0148] Among them, the time series verification task classification head and the masked space recognition task classification head can be implemented using any existing classification network.

[0149] For example, in an optional implementation manner, the time series verification task classification head and / or the masked space recognition task classification head are composed of two fully connected layers.

[0150] For a sample to be tested, the predicted classification results of the time series verification task include the predicted classification results of positive samples and negative samples, where the predicted classification result is "positive" or "negative". "Positive" indicates that the EEG signal of this sample is not inverted in the time dimension, and "negative" indicates that the EEG signal of this sample is inverted in the time dimension. The predicted classification results of the masked space recognition task include the predicted classification results of the samples to be tested after each masking, where the predicted classification result represents the masked area in the samples to be tested after masking.

[0151] S33. Calculate the network loss according to the labels of the time series verification task dataset, the predicted classification results of the time series verification task, the labels of the masked space recognition task dataset, and the predicted classification results of the masked space recognition task, and use the network loss to update the parameters of the original EEG signal classification network to obtain an updated EEG signal classification network.

[0152] In this step, the network loss is calculated using the difference between the prediction results and the labels, and the parameters of the original EEG signal classification network are updated again using the gradient descent method to obtain an updated EEG signal classification network.

[0153] In the embodiments of the present invention, it is found through research that the phenomenon of offset between the data distribution of the training set and the data distribution of the data set during actual testing is the key factor affecting the classification accuracy of the rapid serial visual presentation paradigm. Therefore, the phenomenon that the data distribution during actual testing continuously shifts over time causes the original electroencephalogram (EEG) signal classification network trained using the training set in the sample data set to no longer be applicable to actual testing. In response to this problem, the inventors in the embodiments of the present invention find through research that the unlabeled test data contains rich data distribution information in the test stage. Making full use of this potential distribution information will help the model applicable to the training set distribution to adapt to the actual test stage. Then, how to update the model, that is, the original EEG signal classification network, through this unlabeled data to make it adapt to the distribution during actual testing is the key point studied in the embodiments of the present invention.

[0154] In this step, for each sample to be tested, the model parameters are updated using the above three steps. The whole process establishes a feedback mechanism from the unlabeled sample to be tested to the model parameters through self-supervised learning, thereby updating the originally trained model. For this purpose, considering that EEG signals contain rich temporal and spatial information, the embodiments of the present invention respectively design two self-supervised tasks, namely a temporal verification task and a masked spatial recognition task, to extract temporal and spatial information. These two self-supervised tasks automatically create labels from the unlabeled samples to be tested and can adjust the feature extractor under distribution shift. The temporal verification task utilizes the temporal correlation of EEG signals to extract temporal dimension information by judging whether the sample to be tested is inverted in the temporal dimension. The masked spatial recognition task randomly masks some electrodes in specific brain regions using noise such as Gaussian noise and captures the inherent spatial relationships of different brain regions by identifying the masked regions. The temporal verification task and the spatial masking recognition task respectively help the EEG signal classification task corresponding to the original data (sample to be tested) learn relevant temporal information and spatial information, and through the mutual promotion of the two tasks during the continuous iterative learning process, further adapt to the distribution shift generated during actual testing and improve the classification ability of the model.

[0155] S4. Classify the sample to be tested using the updated EEG signal classification network to obtain the corresponding classification result.

[0156] This step can be understood as model inference, which is the process of realizing EEG signal classification.

[0157] Specifically, inputting the sample to be tested into the updated electroencephalogram signal classification network can obtain its classification result, which is a vector with a dimension of 2. The values therein successively represent the probabilities of no target and the target being a vehicle in the sample to be tested. For example, if the classification result is (0.7, 0.1), it means that the possibility of no target in the image corresponding to the sample to be tested is 0.7, and the possibility of the target being a vehicle is 0.1. Then, according to the magnitudes of the probability values, the final classification result can be determined as no target.

[0158] In an optional implementation manner, the sample to be tested can also be a test sample in the test set. For this implementation manner, first, use the test set to obtain the trained original electroencephalogram signal classification network, and then perform steps S1 to S4 on each test sample in the test set to obtain the classification result of each test sample.

[0159] Furthermore, all classification results obtained for the test set in the above manner can be statistically analyzed to obtain the recognition accuracy rate on the test set, which is used to evaluate the effectiveness of the algorithm or further guide the optimization and adjustment of model parameters, etc.

[0160] In the method for online adaptive classification of electroencephalogram signals based on self-supervised learning provided by the embodiments of the present invention, a sample data set is obtained from the electroencephalogram signals obtained by a sample subject through a rapid serial visual presentation experiment in advance to train a preset network, and a trained model, that is, the original electroencephalogram signal classification network, is obtained. During actual testing, for each sample to be tested obtained after preprocessing the electroencephalogram signals collected from the subject to be tested under rapid serial visual presentation, first construct two self-supervised tasks including a temporal verification task and a masked space recognition task, and obtain a temporal verification task data set and a masked space recognition task data set; then, based on the temporal verification task data set and the masked space recognition task data set, update the parameters of the original electroencephalogram signal classification network to obtain an updated electroencephalogram signal classification network; finally, use the updated electroencephalogram signal classification network to classify the sample to be tested to obtain the corresponding classification result.

[0161] Since the unlabeled sample to be tested presented during actual testing provides information on its data distribution. The method of the embodiments of the present invention transforms a single unlabeled sample to be tested into a self-supervised learning problem, including a temporal verification task and a masked space recognition task. By using the self-supervised learning of the sample to be tested, the parameters of the model trained using test data are updated before prediction to fully extract the distribution information of the electroencephalogram signals during actual testing, and the parameters of the feature extractor are updated during actual testing, thereby reducing the distribution difference between the electroencephalogram signals during training and actual testing, enabling the updated model to fully adapt to the data distribution of the samples during actual testing, and thus improving the classification performance of electroencephalogram signals.

[0162] Compared with the solutions to the traditional electroencephalogram (EEG) signal distribution shift problem, the method of the embodiment of the present invention does not predict the distribution change of EEG data between training data and test data. Instead, it learns from the samples to be measured during actual testing. By making full use of the data signals of the samples to be measured during actual testing, it can more efficiently solve the distribution shift problem, and the training set data is not used to solve the distribution shift problem during the actual testing process. The embodiment of the present invention can solve the distribution shift problem, improve the classification performance of EEG signals in the rapid serial visual presentation paradigm, and can be used in services in multiple fields such as medical treatment, having high application value.

[0163] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An online adaptive classification method for electroencephalogram signals based on self-supervised learning, characterized in that, Including: Obtain a sample to be tested; wherein, the sample to be tested is obtained by preprocessing the electroencephalogram signal collected from the subject to be tested under rapid serial visual presentation; Construct datasets corresponding to the sample to be tested for two self-supervised tasks respectively, the two self-supervised tasks including a temporal verification task and a masked space recognition task; For the temporal verification task, use the sample to be tested as a positive sample, and obtain a negative sample by performing an exchange operation on different parts of the sample to be tested in the time dimension. The negative sample with the corresponding label and the positive sample together constitute the temporal verification task dataset; For the masked space recognition task, according to the preset correspondence between brain regions and electrodes, divide the sample to be tested into a preset number of regions, and each time mask the electroencephalogram signal of a non-repeated region in the sample to be tested with a preset noise to obtain the masked sample to be tested with a label for this time; the masked samples to be tested obtained each time together constitute the masked space recognition task dataset; wherein, the label of the masked sample to be tested obtained each time is the same as the region number of the electroencephalogram signal region masked by the sample to be tested for this time; Based on the temporal verification task dataset and the masked space recognition task dataset, update the parameters of the pre-trained original electroencephalogram signal classification network to obtain an updated electroencephalogram signal classification network; wherein, the original electroencephalogram signal classification network is obtained by training a preset network using a sample dataset, and the sample dataset is obtained based on the electroencephalogram signals obtained from multiple sample subjects in rapid serial visual presentation experiments; Use the updated electroencephalogram signal classification network to classify the sample to be tested to obtain the corresponding classification result.

2. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, wherein The obtaining a negative sample by performing an exchange operation on different parts of the sample to be tested in the time dimension includes: Divide the sample to be tested into two parts, front and back, in the time dimension; Exchange the electroencephalogram signals corresponding to the front and back parts respectively to obtain the negative sample corresponding to the sample to be tested.

3. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, characterized in that The preset correspondence between brain regions and electrodes includes a preset number of brain regions and the names of multiple electrodes corresponding to each brain region for collecting electroencephalogram signals.

4. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, wherein The preset noise includes Gaussian noise.

5. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, wherein The updating the parameters of the pre-trained original electroencephalogram signal classification network based on the temporal verification task dataset and the masked space recognition task dataset to obtain an updated electroencephalogram signal classification network includes: Input the temporal verification task dataset and the masked space recognition task dataset into the original electroencephalogram signal classification network at the same time to obtain the features of the temporal verification task and the features of the masked space recognition task respectively; Input the features of the temporal verification task into the pre-constructed temporal verification task classification head to obtain the predicted classification result of the temporal verification task; and input the features of the masked space recognition task into the pre-constructed masked space recognition task classification head to obtain the predicted classification result of the masked space recognition task; Calculate the network loss based on the labels of the temporal verification task dataset, the predicted classification results of the temporal verification task, the labels of the masked space recognition task dataset, and the predicted classification results of the masked space recognition task, and update the parameters of the original EEG signal classification network using the network loss to obtain the updated EEG signal classification network.

6. The online adaptive classification method for EEG signals based on self-supervised learning according to claim 5, characterized in that The temporal verification task classification head and / or the masked space recognition task classification head consists of two fully connected layers.

7. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, wherein The process of obtaining the sample dataset includes: Conduct a rapid serial visual presentation experiment on the selected sample subjects, and collect the electroencephalogram signals of each sample subject under preset experimental conditions; Preprocess the collected electroencephalogram signals; Construct a sample dataset from all the preprocessed electroencephalogram signals, and divide the sample dataset into a training set and a test set according to a preset ratio.

8. The online adaptive classification method of EEG signals based on self-supervised learning according to claim 1 or 7, characterized in that For any electroencephalogram signal, the preprocessing process includes: According to the timestamps of the occurrences of each stimulus in the rapid serial visual presentation corresponding to the electroencephalogram signal, segment the collected electroencephalogram signal to obtain multiple data segments; where each occurrence of an image in the image sequence corresponding to the rapid serial visual presentation corresponds to a stimulus; Filter each data segment; Perform downsampling on each filtered data segment; Perform normalization on each downsampled data segment.

9. The online adaptive classification method for electroencephalogram signals based on self-supervised learning according to claim 1, characterized in that The preset network includes the EEGNet network.

Images