An event-related potential recognition method based on adversarial convolutional neural network

Abstract

The present invention provides an event-related potential identification method based on an adversarial convolutional neural network, comprising: collecting electroencephalogram (EEG) signals to be identified; constructing a feature extraction network comprising multiple convolutional layers; after adversarial learning, the feature extraction network extracts a feature vector group of the EEG signal in terms of time, space, and frequency dimensions related to the event-related potential layer by layer; constructing a classifier comprising two softmax layers; identifying the feature vector group of the event-related potential by the classifier to obtain an event-related potential identification result; wherein, during adversarial learning of the feature extraction network, the feature vector group extracted in each iteration is clustered, the proportion of misclassified samples after clustering is calculated, and this loss is back-propagated to the feature extraction network with a certain weight in an adversarial manner for learning. This method improves the accuracy of ERP identification and has good stability.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to an event-related potential recognition method based on an adversarial convolutional neural network. Background Art

[0002] Brain-Computer Interfaces (BCIs) can directly connect to external devices using brain signals, enabling information exchange between the two devices. Among various brain signals, electroencephalogram (EEG) signals are currently being widely studied due to their high temporal resolution, non-invasiveness, low cost, and safety.

[0003] Based on EEG signals, event-related potentials (ERPs) are specialized brain-evoked potentials. They exploit brain potentials evoked by multiple or diverse stimuli by intentionally assigning them specific psychological significance. To extract ERP features from EEG signals, repeated stimulus segments are cut and averaged. Since the EEG waveform that serves as the background for the ERP has no fixed relationship with the stimulus, the ERP components contained within it exhibit time-locked characteristics and similar structures after each stimulus. When superimposed, the ERP amplitude increases proportionally with the number of superpositions. ERPs reflect neurophysiological changes in the brain during cognitive processes. Also known as cognitive potentials, they refer to brain potentials recorded from the surface of the skull when people engage in cognitive processing. ERP technology can isolate neurocognitive operations that unfold rapidly following a stimulus, response, or other event. Currently, many ERP components, such as the P300, N170, and N400, have been identified and validated as measures of sensory, cognitive, emotional, and motor processes. In recent years, thanks to the development of relatively inexpensive EEG recording systems and commercial and open-source software packages for processing ERP data, ERP technology has been widely used by researchers. Despite significant progress in various areas of BCI technology, several issues remain that hinder the widespread deployment and adoption of EEG-based brain-computer interfaces. For example, because EEG signals are non-invasive, they have a low signal-to-noise ratio. Furthermore, ERP signals are highly specific to each individual, and the differences between individuals in the same signal can become increasingly pronounced over time. While some aspects of EEG recording and processing have become relatively standardized, many other studies vary significantly between laboratories and even within laboratories. For example, the P3 component has been measured using the oddball paradigm in thousands of studies, but task parameters, recording setups, and data processing methods vary widely across studies. As BCI technology advances into new phases, the demand for robust feature extraction techniques will only increase.

[0004] ERP components are highly specific to each individual, and over time, the differences in the same signal between different individuals may gradually increase. Furthermore, within individuals, over time or due to factors such as attention, the ERP components of different segments may also vary. Traditional ERP component analysis can only reflect the overall impact of specific factors on the subject and cannot analyze specific time periods, affecting practical applications in fields such as clinical and criminal investigation. Deep learning algorithms can address this shortcoming, but existing network models designed for ERP components are insufficient, and performance comparisons of various models are still insufficient, resulting in relatively average model stability and accuracy. Research on the applicability of models to various types of paradigms, as well as on the model's effectiveness within and across subjects, is insufficiently comprehensive. Summary of the Invention

[0005] To solve the above problems, the present invention provides an event-related potential recognition method based on an adversarial convolutional neural network, and proposes a deep learning model that is universal within and across subjects under various paradigm conditions.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] An event-related potential recognition method based on an adversarial convolutional neural network includes the following steps:

[0008] collecting electroencephalogram signals to be identified;

[0009] A feature extraction network is constructed based on a convolutional neural network. After adversarial learning, the feature extraction network extracts the feature vector groups of event-related potentials of the EEG signal layer by layer in the time, space and frequency dimensions.

[0010] Constructing a classifier, identifying the feature vector group of the event-related potential through the classifier, and obtaining the component type of the event-related potential in the electroencephalogram signal;

[0011] Among them, when the feature extraction network is adversarially learning, the feature vector groups extracted in each iteration are clustered, the proportion of misclassified samples after clustering is calculated, and backpropagation is performed to the feature extraction network for learning in an adversarial manner.

[0012] Preferably, it also includes:

[0013] The EEG signals were filtered and processed by independent component analysis to remove artifacts.

[0014] Preferably, the feature extraction network includes:

[0015] n1 2D convolutional layers with kernel size (1, 64) are used to extract feature vector groups of EEG signals at different bandpass frequencies;

[0016] n1 2D convolutional layers with kernel size (11, 1) are used to extract feature vector groups in the spatial dimension;

[0017] n2 2D convolutional layers with kernel size (4, 4) are used to synthesize feature vector groups at different times and spaces;

[0018] The depthwise separable convolution layer and the pointwise convolution layer are used to summarize all the previously extracted feature feature groups and merge them into a feature vector group that outputs the event-related potential in the best way;

[0019] Among them, the number of the depth-wise separable convolutional layers is n2, and the convolution kernel size is (1, 1); the number of the point-wise convolutional layers is n2, and the convolution kernel size is (1, 1).

[0020] Preferably, the first softmax layer of the classifier outputs 100-dimensional feature points, and the second layer outputs 2-dimensional feature points.

[0021] Preferably, it also includes:

[0022] The entire network model is fitted using the Adam optimizer by minimizing the classification cross entropy loss function;

[0023] Through multiple training iterations and performing validation stops, save the model weights that produce the lowest validation set loss.

[0024] Preferably, it also includes:

[0025] The entire network model is optimized using the Dropout technology; the dropout probability is set to 0.20 and the batch size is set to 32.

[0026] Beneficial effects of the present invention:

[0027] This paper proposes an event-related potential (ERP) identification method based on an adversarial convolutional neural network. This method improves the accuracy of ERP identification and exhibits excellent stability. It demonstrates good sensitivity to ERP components within subjects and reduced sensitivity across subjects, reducing the error rate in cross-subject ERP identification. This method is more suitable for current application environments and practical requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A network structure diagram of an embodiment of the present invention;

[0029] Figure 2 The results of the intra-subject model analysis of an embodiment of the present invention;

[0030] Figure 3This is the cross-subject model analysis result of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] Example 1

[0033] An event-related potential recognition method based on adversarial convolutional neural network, such as Figure 1 As shown, the following steps are included:

[0034] S1: Collect the EEG signal to be identified.

[0035] S2: Construct a feature extraction network, including multiple convolutional layers; after adversarial learning, the feature extraction network extracts the feature vector group of event-related potentials of the EEG signal in the time, space and frequency dimensions layer by layer.

[0036] S3: Construct a classifier, including two softmax layers; identify the feature vector group of event-related potentials through the classifier to obtain event-related potential recognition results.

[0037] Specifically:

[0038] (1) Pretreatment

[0039] EEG data is 30 channels (FP1, F3, F7, FC3, C3, C5, P3, P7, P9, PO7, PO3, O1, Oz, Pz, CPz, FP2, Fz, F4, F8, FC4, FCz, Cz, C4, C6, P4, P8, P10, PO8, PO4, O2), and the sampling frequency is 256H Z All signals were filtered and then subjected to independent component analysis (ICA) to remove artifacts. Data deemed unacceptable by official assessments were removed. Model testing was performed using the P300 and ERN components of the dataset. The EEG signals eliciting the ERP component in each round were intercepted and used as the dataset for subsequent model training.

[0040] (2) Model structure

[0041] The overall model structure is as follows Figure 1 As shown in Figure 1, to train a deep neural network to predict the target value of each input ERP, a feature extraction network is first applied to the ERP signal to obtain a set of feature vectors. The feature extraction network uses the Rouet-Leduc method to extract continuous signals in the time dimension layer by layer.

[0042] The network consists of multiple convolutional layers. First, n1 2D convolutional layers with kernel sizes of (1, 64) are used to extract feature vectors of the EEG signal at different bandpass frequencies. Then, n1 2D convolutional layers with kernel sizes of (11, 1) are used to extract features in the spatial dimension. Then, n2 2D convolutional layers with kernel sizes of (4, 4) are used to synthesize features across time and space. Finally, the feature groups are summed up and combined into the optimal output using depthwise separable convolutional layers (n2*kernel size of (1, 1)) and pointwise convolutional layers (n2*kernel size of (1, 1)).

[0043] This structural combination is very suitable for EEG signals and can effectively integrate features at different scales (time, space, and frequency). The intra-subject and cross-subject modes can be adapted to the dataset by adjusting the number of convolution kernels. In this example, the intra-subject mode (n1 = 12, n2 = 32) and the cross-subject mode (n1 = 8, n2 = 24) are used.

[0044] To ensure that the extracted features are not biased towards confounding factors, the feature vectors extracted in each round are clustered to assess the degree of confounding factors in the features. The proportion of misclassified samples after clustering is calculated, and this loss is backpropagated to the feature extraction module in an adversarial manner with a certain weight. Thus, the feature extractor learns features that minimize the loss of the predicted classification by using the degree of loss after clustering, while being conditionally independent of confounding factors.

[0045] The classifier consists of two softmax layers. The first layer outputs 100-dimensional feature points, and the second layer outputs the probability that the corresponding sample belongs to one of the two categories. Losses generated by the adversarial module are not directly passed to the classifier, but indirectly affect the classifier parameters through the feature module. After continuous iteration, features are extracted without confounding factors, and the feature vector group retains the features relevant to the classification.

[0046] The model was fitted using the Adam optimizer, minimizing the categorical cross-entropy loss function. 500 training iterations (EPOCH) were run, with validation stopping performed to save the model weights that produced the lowest validation set loss. To help regularize the model and prevent overfitting, dropout was used, with a dropout probability of 0.20. The batch size was set to 32. Further network parameters are shown in Table 1.

[0047] Figure 1 ERPNET network parameters

[0048]

[0049] In this embodiment,

[0050] 1. Configuration of Embodiment

[0051] The EEG data of each round of tasks of the subjects are intercepted. In the intra-subject model part, the model is trained for the data of each subject. The classification effects of different classifiers on the EEG data of each subject are compared. In addition, the performance of the cross-subject model is often lower than that of the intra-subject model on various tasks. Therefore, eliminating the influence of different subjects and improving the stability of the model are also important requirements of the model. Four subjects are divided into a group to ensure that each subject is used as a validation set to evaluate the performance of the model after cross-validation. This embodiment uses intra-subject and cross-subject analysis to train the model. Use four-fold cross-validation and compare the classification effects of different classifiers under different data sets (P300; ERN).

[0052] 2 Effects

[0053] 2.1 Within-subject effects

[0054] For within-subjects model analysis results, see Figure 2 Table 2. CNN-based and DBN-based network models significantly outperformed traditional methods (p < 0.01). In the P300 dataset, CNN-based network models (NET; EEGNet; ShallowConvNet) significantly outperformed DBN-based models (p < 0.01), while DBN-based models showed poor stability. DBN models exhibited significant data bias and were highly sensitive to non-ERP components.

[0055] The ERPNET network model achieved an accuracy of 86.9% in the P300 dataset and 93.65% in the ERN dataset, both significantly outperforming other CNN-based network models (P < 0.01). ERPNET demonstrated greater sensitivity to ERP components compared to other models (p < 0.01; P300: 46.65%, ERN: 53.89%). EEGNet, DeepConvNet, and traditional methods demonstrated significantly better sensitivity to ERP components than ShallowConvNet on both datasets, although ShallowConvNet showed less stable sensitivity. EEGNet demonstrated higher sensitivity than DeepConvNet on the ERN dataset (p = 0.04). No significant differences were observed in sensitivity between EEGNet, DeepConvNet, and traditional methods on the P300 dataset.

[0056] Table 2 shows the results of the within-subject model analysis.

[0057]

[0058] 2.2 Cross-subject effects

[0059] For cross-subject model analysis results, see Figure 3 Table 3. The DBN model exhibits more severe bias (P300 component sensitivity 0.11%; ERN component sensitivity 0%). Furthermore, traditional methods also exhibit significant bias on the ERN dataset, with a sensitivity of 1.44%. CNN-based network models exhibit less data bias, with ERPNET achieving accuracy rates of 74.7% on the P300 dataset and 88.06% on the ERN dataset. On both datasets, the network significantly outperformed the EEGNet model (p < 0.01). Furthermore, on the ERN dataset, it significantly outperformed the DeepConvNet model (p = 0.03). Compared to other CNN-based network models, EEGNet and ShallowConvNet exhibit less stability.

[0060] Table 3 shows the results of cross-subject model analysis.

[0061]

[0062] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An event-related potential recognition method based on an adversarial convolutional neural network, characterized in that: The following steps are involved: collecting electroencephalogram signals to be identified; Construct a feature extraction network based on convolutional neural network; After adversarial learning, the feature extraction network extracts the feature vector groups of event-related potentials of the EEG signal layer by layer in the time, space and frequency dimensions. Constructing a classifier, identifying the feature vector group of the event-related potential through the classifier, and obtaining the component type of the event-related potential in the electroencephalogram signal; Among them, when the feature extraction network is adversarially learning, the feature vector groups extracted in each iteration are clustered, the proportion of misclassified samples after clustering is calculated, and backpropagation is performed to the feature extraction network in an adversarial manner for learning; The feature extraction network includes the following connected in sequence: n1 2D convolutional layers with kernel size (1, 64) are used to extract feature vector groups of EEG signals at different bandpass frequencies; n1 2D convolutional layers with kernel size (11, 1) are used to extract feature vector groups in the spatial dimension; n2 2D convolutional layers with kernel size (4, 4) are used to synthesize feature vector groups at different times and spaces; The depthwise separable convolution layer and the pointwise convolution layer are used to summarize all the previously extracted feature feature groups and merge them into a feature vector group that outputs the event-related potential in the best way; Among them, the number of the depth-wise separable convolutional layers is n2, and the convolution kernel size is (1, 1); the number of the point-wise convolutional layers is n2, and the convolution kernel size is (1, 1).

2. The event-related potential recognition method based on adversarial convolutional neural network according to claim 1, characterized in that: Also includes: The EEG signals were filtered and processed by independent component analysis to remove artifacts.

3. The event-related potential recognition method based on adversarial convolutional neural network according to claim 1, characterized in that: The first softmax layer of the classifier outputs 100-dimensional feature points, and the second layer outputs 2-dimensional feature points.

4. The event-related potential recognition method based on adversarial convolutional neural network according to claim 1, characterized in that: Also includes: The entire network model is fitted using the Adam optimizer by minimizing the classification cross entropy loss function; Through multiple training iterations and performing validation stops, save the model weights that produce the lowest validation set loss.

5. The event-related potential recognition method based on adversarial convolutional neural network according to claim 1, characterized in that: Also includes: The entire network model is optimized using the Dropout technology; the dropout probability is set to 0.20 and the batch size is set to 32.

Images