Electroencephalogram feature interpretability method based on neural network decoding

Abstract

The present application relates to the field of brain-computer interface and computer science, and provides a neural network feature interpretability method suitable for P300 signals and motor imagery signals, and a visualization method for specific EEG events. The present application is based on a neural network decoding EEG feature interpretability method, and the steps are as follows: 1) sending the original EEG data obtained through acquisition to a neural network for training after pre-processing; 2) training the neural network and saving the optimal neural network test model; 3) using the trained neural network model to perform category prediction on the EEG data to be tested, marking the EEG test times predicted correctly by the neural network, and performing interpretability analysis on the neural network model. The present application is mainly applied to brain-computer interface design occasions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of brain-computer interface and computer science, in particular to a brain electrical feature interpretability method based on neural network decoding. BACKGROUND

[0002] EEG-based brain-computer interface plays an important role in motor rehabilitation, brain function regulation, and mental illness detection. EEG signals can often be obtained by non-invasive methods. Compared with methods such as magnetic resonance imaging (MRI) and computer tomography (CT), EEG has the advantages of low cost and high temporal resolution. However, EEG signals also have the disadvantages of low spatial resolution and low signal-to-noise ratio, which puts higher requirements on the decoding of EEG.

[0003] With the development of neural network related technologies, EEG decoding technology based on neural networks has also been widely applied. Compared with the method of manually extracting features to decode EEG, the EEG decoding method based on neural networks does not require the assistance of prior knowledge and relies on the powerful fitting ability of neural networks to automatically learn features with discriminability for classification. In the decoding of EEG signals such as motor imagery and P300 (an event-related potential component produced during decision-making), by setting appropriate neural network architecture, the decoding technology based on neural networks has better feature extraction ability than the manual feature extraction method, which is manifested in that the neural network method can obtain higher classification accuracy than the manual feature extraction method.

[0004] A problem worth noting is that neural networks are black box models, and the classification features learned automatically are not known, which to some extent limits the application of neural network methods in EEG. Although feature visualization methods based on neural networks have mature applications in machine vision, EEG signals and images are different in nature. Industry personnel can provide strong prior knowledge about the information contained in images, which can be used as a criterion for judging the reasonableness of neural network feature interpretation. However, even experienced EEG practitioners cannot point out the feature information contained in the waveform of EEG signals. This unknowability seriously limits the application of neural networks in EEG decoding.

[0005] The existing EEG feature explainability method based on neural network decoding visualizes the neural network kernel function information to reflect the features learned by the neural network. This explainability method is inherited from the neural network visualization method in machine vision. The papers published in high-level foreign journals, such as “EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces” and “Deep learning with convolutional neural networks for EEG decoding and visualization”, both adopt this explainability method. However, this method has the following two problems:

[0006] First, the number of neural network kernel functions is large, and the learned features of each kernel function are different, so the visualized kernel function reflects a complex and diverse result, which cannot provide an accurate evaluation of the visualized result of each kernel function.

[0007] Second, the method of visualizing the kernel function cannot reflect the differences between different categories of EEG features. Once the neural network is trained, the kernel function is fixed. The trained neural network is a mapping of the data in the entire data set, or even under a certain data distribution, so it is difficult to extract the feature information of a single category from it.

[0008] In summary, the method of visualizing the kernel function is not suitable for event-driven EEG signals. SUMMARY

[0009] To overcome the shortcomings of the prior art, the present application aims to provide a neural network feature explainability method suitable for P300 signals and motor imagery signals, and a visualization method for specific EEG events. To this end, the technical solution adopted by the present application is as follows:

[0010] 1) The original EEG data collected is preprocessed and then sent to the neural network for training;

[0011] 2) Train the neural network and save the optimal neural network test model;

[0012] 3) Use the trained neural network model to predict the categories of the EEG data to be tested, mark the EEG trials correctly predicted by the neural network, and perform explainability analysis on the neural network model.

[0013] The detailed steps of step 1) are as follows:

[0014] a) Band-pass filtering

[0015] b) down-sampling

[0016] c) trial division

[0017] d) data normalization.

[0018] When the EEG signal in step 3) is an event-related potential signal P300, the interpretability analysis is divided into neural network feature visualization, artificial feature extraction visualization based on neuroscientific prior knowledge, and comparison analysis of neural network feature visualization and artificial feature extraction visualization, and the specific steps are as follows:

[0019] a) The steps of neural network learned feature visualization are as follows:

[0020] i. For P300 signal, the object of neural network learned feature visualization is the EEG data of all participants to be tested, and the first n trials with the highest prediction confidence for each participant and each category are obtained, which are called significant trials, denoted as where subscript i and j represent the number of participants and the number of categories, respectively, N represents the number of significant trials corresponding to each participant and each category, denoted as dim = 0, indicating the 0th dimension; C represents the number of leads, denoted as dim = 1; T represents the number of sampling points contained in each trial, denoted as dim = 2;

[0021] ii. Using class activation mapping CAM, using the significant trials of all categories in i as input, the class activation mapping values corresponding to the spatial feature extraction layer and the time domain feature extraction layer of the trained neural network model are obtained, denoted as and

[0022] iii. For T ij and S ij , the arithmetic mean of all trials is calculated, dim = 0, to eliminate the influence of the number of trials on the class activation mapping value, denoted as and

[0023] iv. For each participant and each category, the spatial dimension arithmetic mean of T' ij is calculated, dim = 1, denoted as The superposition of T" ij for all participants is performed to obtain the time domain class activation mapping value corresponding to each category, denoted as

[0024] v. T" j is mapped into a heat map according to the value size, which is used as the visualization result of the neural network learned feature in the time domain for each category;

[0025] vi. Calculate S′ for all participants in each category. ij The arithmetic mean, the result is denoted as This is to eliminate the influence of the number of participants on the visualization results of neural network features for each category, while recording T″. j The time-domain coordinate t corresponding to the maximum value j , search t j Time S′ j The corresponding value is denoted as

[0026] vii. Divide the different categories of s′ j The neural network features are visualized in the spatial domain by displaying them in the manner of brain topography.

[0027] b) The steps for manually extracting and visualizing features based on prior neuroscience knowledge are as follows:

[0028] i. For each participant and each category in (a)-i, the significant trials D ij Calculate the arithmetic mean of all trials where dim = 0, resulting in an EEG dataset independent of the number of trials, denoted as .

[0029] ii. Calculate D′ for all participants in each category. ij The arithmetic mean, denoted as From D′ j Select the value corresponding to lead Cz d j That is, the amplitude of the EEG signal corresponding to lead Cz under different events, plotted as d j The time-amplitude curves were calculated, and the difference in amplitude between normal events and oddball events at the corresponding time in lead Cz was also calculated.

[0030] iii. Choose T″ from a)-vi j The time coordinate t corresponding to the maximum value j Calculate and plot D′ j In t j Brain topography at time

[0031] c) Comparative analysis of neural network feature visualization and manually extracted feature visualization:

[0032] iv. Comparing the oddball event neural network feature heat map and the Cz lead time-amplitude curve in a)-v and b)-ii, if the significant region in the neural network feature heat map corresponds to the time associated with the peak value in b)-ii, and the peak value refers to the maximum positive peak value of the oddball event or the maximum value of the difference between the normal event and the oddball event, it is considered that the positioning of the neural network learned features in the time domain is reasonable; for the normal event EEG signal, if the above time is distinguishable from the time determined in the oddball event, and the neural network feature heat map and some peak values in b)-ii have a corresponding relationship, some peak values include the peak value of the normal event and the peak value of the difference between the normal event and the oddball event, it is considered that the reaction of the neural network learned features to the normal event is also reasonable, if the neural network learned features and the above description have large differences, it is considered that the neural network cannot reasonably position P300 in the time domain;

[0033] v. In the oddball event, the time at which the maximum value of the neural network feature heat map is located is t0, that is, the time corresponding to class j in a)-vi; the time at which the maximum value of the neural network feature heat map in the normal event is located is t1, and the difference between the EEG brain topography in b)-iii and the neural network feature brain topography in a)-vii at these two times is observed; if the EEG brain topographies of different classes in a)-vii and b)-iii at t0 have large differences, it indicates that the neural network features and the neural network features at t0 have good spatial distinguishability; the importance of each lead in classification can be further explained by the brain topography corresponding to the oddball event in a)-vii.

[0034] In step 3), for the event-related desynchronization ERD / event-related synchronization ERS signal represented by the motor imagery signal, the explainability analysis of the neural network features is the spatial visualization of the neural network learned features, and the specific steps are as follows:

[0035] a) For motor imagery data, the object of neural network feature explainability is the EEG data of a single participant, and a trained neural network model is used to test the EEG data of a participant to obtain the first n test times of each class with the highest confidence and correct prediction, which is denoted as significant test times

[0036] b) Using class activation mapping, using the significant test times of all classes in a) as input, obtaining the class activation mapping values corresponding to the spatial feature extraction layer of the trained neural network model and the class activation mapping values corresponding to the time domain feature extraction layer

[0037] c) For T j Performing the arithmetic mean with spatial dimension dim = 1 yields a spatially independent, temporally correlated neural network feature category activation mapping value, denoted as . Select T′ j The first m maximum values ​​of each trial are recorded, and their corresponding time points are denoted as the significance time set S. t ;

[0038] d) Using the saliency time set S from c) t For S j By creating an index with time dimension dim=2, we obtain... Where T′ represents the length of the significant time set;

[0039] e) Iterate through S′ according to the trial dim = 0 in each category. j And for each category and each trial, S′ j Summing by dim=2 in the time dimension yields the mapping value of the neural network features for each category and each trial in the spatial dimension. and the obtained Probabilistic approach;

[0040] f) According to the superposition of trials s′ j ,get The data is displayed in the form of brain topography, resulting in a spatial visualization of different categories of neural network features.

[0041] The features and beneficial effects of this invention are:

[0042] This invention, based on prior knowledge in neuroscience and neural network visualization techniques in computer science, proposes a method for interpreting neural network features applicable to P300 signals and motion imagery signals. This invention not only interprets the features learned by neural networks based on prior knowledge in neuroscience, but also solves the problems of poor interpretability and inability to visualize specific EEG events in existing neural network feature visualization methods, thus greatly promoting the application of neural networks in EEG. Attached image description:

[0043] Figure 1 This is a diagram of the neural network architecture used for decoding EEG signals in the embodiment.

[0044] Figure 2 This is the location of the P300 signal features in the embodiment.

[0045] Figure 3 This is the spatial localization of neural network features in motion image signals in the embodiment. Detailed Implementation

[0046] The application is based on prior knowledge in neuroscience and neural network visualization technology in computer science, and proposes a neural network feature explainability method suitable for event-related EEG signals. Compared with existing neural network visualization methods, the application not only can explain the features learned by the neural network based on prior knowledge in neuroscience, but also solves the problems of poor explainability and inability to provide visualization for specific EEG events of existing neural network feature visualization methods, which can greatly promote the application of neural networks in EEG.

[0047] The application aims to provide an EEG feature explainability method based on neural network decoding. The technical solution is as follows:

[0048] 1. The original EEG data collected needs to be sent to the neural network for training after several steps or all steps:

[0049] a) Band-pass filtering

[0050] b) Down-sampling

[0051] c) Trial segmentation

[0052] d) Data normalization

[0053] Among them, band-pass filtering and trial segmentation are necessary steps.

[0054] 2. Train the neural network and save the optimal neural network test model.

[0055] 3. Use the trained neural network model to make category prediction on the EEG data to be tested. Label the EEG trials predicted correctly by the neural network.

[0056] 4. For event-related potential (ERP) signals represented by P300, the explainability analysis includes neural network feature visualization, artificial extracted feature visualization based on prior knowledge in neuroscience, and comparison analysis of neural network feature visualization and artificial extracted feature visualization, and the specific steps are as follows:

[0057] a) The steps of neural network learned feature visualization are as follows:

[0058] i. For P300 signals, the object of neural network learned feature visualization is the EEG data of all participants to be tested, and the first n (such as n=5) trials with the highest prediction confidence for each participant and each category predicted correctly are obtained, which are called significant trials and are denoted as where subscript i and j represent the number of participants and the number of categories respectively, N represents the number of significant trials corresponding to each participant and each category, dim = 0 represents the 0th dimension; C represents the number of leads, dim = 1; T represents the number of sampling points contained in each trial, dim = 2.

[0059] ii. Using the category activation map CAM, taking the significant trials of all categories in i as input, the category activation map value corresponding to the spatial feature extraction layer of the trained neural network model (denoted as ) and the category activation map value corresponding to the time domain feature extraction layer (denoted as ) are obtained. Wherein i, j, N, C, T have the same meaning as a)-i.

[0060] iii. For each participant, the T ij and S ij of each category are calculated, and the arithmetic mean of all trials (dim = 0) is calculated to eliminate the influence of the number of trials on the category activation map value, respectively and

[0061] iv. For each participant, each category, the arithmetic mean value of T' ij in the spatial dimension (dim = 1) is obtained The T" ij of all participants is superimposed to obtain the time domain category activation map value corresponding to each category

[0062] v. T" j is mapped into a heat map according to the value size, which is used as the visualization result of the neural network learned features of each category in the time domain.

[0063] vi. The arithmetic mean value of S' ij of all participants to be tested in each category (the result is denoted as ) is calculated to eliminate the influence of the number of participants on the visualization result of the neural network features of each category. At the same time, the time domain coordinate t j corresponding to the maximum value of T" j is recorded, and the value corresponding to S' j at t j is searched, denoted as

[0064] vii. s' j of different categories is displayed in the form of brain topography to obtain the visualization result of the neural network features in the spatial domain.

[0065] b) The artificial extracted feature visualization steps based on prior knowledge of neuroscience are as follows:

[0066] i. For each participant and each category in 4-a)-i, the significant trials D ij Calculate the arithmetic mean of all trials (dim=0) to obtain an EEG dataset independent of the number of trials, denoted as .

[0067] ii. Calculate D′ for all participants in each category. ij The arithmetic mean, denoted as From D′ j Select the value corresponding to lead Cz d j That is, the amplitude of the EEG signal corresponding to lead Cz under different events, plotted as d j The time-amplitude curve reflects the fluctuation of the amplitude of EEG signals of two types, normal events and oddball events, at lead Cz over time. It also calculates the difference in amplitude between normal events and oddball events at the corresponding time in lead Cz.

[0068] iii. Choose T′ from 4-a)-vi j The time coordinate t corresponding to the maximum value j Calculate and plot D′ j In t j Brain topography at time It reflects the response of different leads to a certain type of EEG signal at a specific time.

[0069] c) The specific steps for comparing and analyzing neural network feature visualization and manually extracted feature visualization are as follows:

[0070] i. Compare 4-a)-v and 4-b)-ii, and compare the neural network feature heatmap of the oddball event with the time-amplitude curve of lead Cz. If the time corresponding to the significant region in the neural network feature heatmap is correlated with the time corresponding to the relevant peak in 4-b)-ii (referring to the maximum positive peak of the oddball event or the maximum difference between the normal event and the oddball event), then the localization of the features learned by the neural network in the time domain is considered reasonable. For the EEG signal of the normal event, if the above time is distinguishable from the time determined in the oddball event, and the neural network feature heatmap corresponds to certain peaks in 4-b)-ii (including the peak of the normal event and the peak of the difference between the normal event and the oddball event), then the response of the features learned by the neural network to the normal event is considered reasonable. If the features learned by the neural network differ significantly from the above description, then the neural network cannot reasonably localize P300 in the time domain.

[0071] ii. Let t0 be the time when the maximum value of the neural network feature heatmap occurs in the oddball event, which corresponds to the time when category j in 4-a)-vi is the oddball event; let t1 be the time when the maximum value of the neural network feature heatmap occurs in the normal event. Observe the differences between the brain topography maps of EEG signals in 4-b)-iii and the neural network feature brain topography maps in 4-a)-vii at these two times. If there are significant differences between the EEG brain topography maps of different categories in 4-a)-vii and 4-b)-iii at time t0, this indicates that both neural network features and neural network features have good spatial distinguishability at time t0. The brain topography map corresponding to the oddball event in 4-a)-vii can further explain the importance of each lead in classification.

[0072] 5. For ERD / ERS (Event-Related Desynchronization / Event-Related Synchronization) signals, represented by motion imagery signals, the interpretability analysis of neural network features involves spatial visualization of the features learned by the neural network. The specific steps are as follows:

[0073] a) For motion imagery data, the data distribution varies significantly among different participants. The interpretability of neural network features lies in the EEG data of individual participants. A trained neural network model is used to test a participant's EEG data, obtaining the top n (e.g., n=10) trials with the highest correct predictions and confidence levels for each category for that participant; these are denoted as significant trials. The meanings of j, N, C, and T are as described above.

[0074] b) Using class activation mapping, with the salient trials of all classes in 5-a) as input, obtain the class activation mapping values ​​corresponding to the spatial feature extraction layer of the trained neural network model. Category activation mapping values ​​corresponding to the temporal feature extraction layer

[0075] c) For T j Performing an arithmetic mean along the spatial dimension (dim=1) yields a spatially independent, temporally correlated neural network feature category activation mapping value, denoted as . Select T′ j For each trial, the first m (e.g., m=10) maximum values ​​are recorded, and their corresponding time points are denoted as the significance time set S. t .

[0076] d) Using the saliency time set S from i) t For S j Indexing along the time dimension (dim=2), we get Where T′ represents the length of the saliency time set.

[0077] e) traverse S' by trial (dim=0) in each class j and S' by trial (dim=0) in each class j sum up in time dimension (dim=2) to get the mapping value of neural network feature in spatial dimension for each trial and each class and get the mapping value of neural network feature in spatial dimension for each trial and each class and get the mapping value of neural network feature in spatial dimension for each trial and each class

[0078] f) stack s' by trial j and get the mapping value of neural network feature in spatial dimension for each trial and each class and get the mapping value of neural network feature in spatial dimension for each trial and each class

[0079] The present application is to visualize and explain the trained neural network. Steps 1-3 represent the training process of the neural network, so as to obtain the trained network model. Step 4 illustrates the use of the trained neural network model to perform explainable analysis on P300 signals (the error-related negativity signal in the example is a P300 signal). In the explainable analysis, some samples that perform well in the neural network model need to be selected first, and the selection of the samples corresponds to step 4-a)-i. These samples are called significant trials. The remaining part of step 4-a) describes the visualization method of the neural network model in these significant trials. Step 4-b) describes the visualization analysis of these significant trials using the manually extracted features in neuroscience. Step 4-c) is the comparative analysis of the visualization results in steps 4-a) and 4-b), which achieves the purpose of explainability.

[0080] Because the experimental object of step 5 is motor imagery signal and the experimental object of step 4 is P300 signal, although both are EEG signals, because the mechanism of generation is different, the characteristics and analysis methods are different. Because P300 signal has the characteristics of time-locking and phase-locking, and the difference between different participants is small, therefore, in step 4, the visualization of the neural network model and the visualization of the manually extracted features are comprehensively analyzed. Motor imagery signal does not have the characteristics of phase locking and the difference between different participants is large, so in step 5, only the explainability analysis of the neural network features of motor imagery signal is performed.

[0081] The application will be further described in connection with the embodiments and the accompanying drawings, but not as the basis for limiting the application.

[0082] An embodiment: a neural network-based electroencephalogram (EEG) feature explainable method, the method comprising the following four steps: EEG data acquisition, neural network training and testing, class activation mapping and EEG feature localization. The EEG signal in this example refers to P300 signal and motor imagery signal, wherein P300 uses the data in the P300-Speller (P300 spelling paradigm) public dataset in Kaggle competition https: / / www.kaggle.com / c / inria-bci- challenge ), hereinafter referred to as the error-related negativity dataset; the data used for motor imagery is the motor imagery public dataset BCI4-2a in BCI (brain-computer interface) competition https: / / www.bbci.de / competition / iv / ). The classification goal of the error-related negativity dataset is to judge whether the selected item is correct by analyzing the brain signals of the participants after receiving feedback. The error-related negativity dataset requires training a cross-participant model, that is, using a specified participant for training and using other participants to predict the model. Therefore, the test set in this embodiment uses the full data of the participant to be tested. The classification goal of the BCI4-2a dataset is to judge the category of the motor imagery of the participant by decoding the EEG signal, which is one of left hand, right hand, both feet or tongue. There are a total of 9 participants in the dataset, and the embodiment uses two participants A03 (participant number, same below) and A06.

[0083] For the data in the error-related negativity dataset, a 1-40Hz band-pass filter is used for filtering. For the data in the BCI4-2a dataset, a 4-38Hz band-pass filter is used for filtering. Whether it is the error-related negativity dataset or the BCI4-2a dataset, the initial EEG data of each participant is 2-dimensional (time dimension and space dimension) data, and the original EEG data needs to be segmented to obtain a data form like (N, C, T), where N represents the number of trials, dim=0, C represents the number of leads, dim=1, and T represents the number of sampling points contained in each trial, dim=2. For the BCI4-2a dataset, some preprocessing (such as data normalization) is also needed to further reduce the interference of noise on feature extraction.

[0084] Figure 1The diagram illustrates the neural network architecture used for EEG decoding in this embodiment. This neural network model contains two convolutional layers. The first convolutional layer is a temporal convolutional layer with a kernel function shape of (24, 1, 75), where 24 represents the number of kernels, 1 represents the length of the kernel in the spatial domain, and 75 represents the length of the kernel in the temporal domain. The second convolutional layer is a spatial convolutional layer with a kernel function shape of (9, C, 1), where C represents the number of leads. To reduce the number of parameters in the neural network, both the temporal and spatial convolutional layers use separable convolutions. If the neural network architecture for decoding EEG and... Figure 1 If the two are different, the spatial convolutional layer can be the layer that performs the last convolution operation on the spatial domain, and the temporal convolutional layer can be the temporal convolutional layer that precedes the aforementioned spatial convolutional layer.

[0085] Using the error-correlated negative wave dataset and the BCI4-2a dataset respectively, Figure 1 The neural network model is trained using a method that follows the general neural network training process. Early-stopping and other strategies are employed to prevent overfitting during training and to obtain a well-trained neural network model.

[0086] The steps for interpreting the features learned by the neural network for the error-correlated negative wave dataset are as follows:

[0087] 1. Select all participants from the error-related negative wave dataset, and use the trained neural network to predict the EEG data of each participant sequentially. For each participant, obtain the top 5 trials with the highest prediction confidence for each category, and denote them as significant trials. Where i represents the participant's ID; j represents the category ID, j=0 represents the oddball event, corresponding to the EEG signal of the error event, j=1 represents the normal event, corresponding to the EEG signal of the correct event; N=5 represents the number of significant trials for each participant in each category, denoted as dim=0; C=56 represents the number of leads, denoted as dim=1; T=250 represents the number of sampling points in each trial, denoted as dim=2.

[0088] 2. Using the Eigen-CAM technique in class activation mapping (reference: "Eigen-cam: Classactivation map using principal components"), with the salient order of each class in step 1 as input, the class activation map values ​​corresponding to the spatial feature extraction layer of the trained neural network model are obtained. Category activation mapping values ​​corresponding to the temporal feature extraction layer

[0089] 3. For T ij , take the arithmetic mean of all trials (dim=0) to obtain a class activation map value corresponding to the time feature extraction layer independent of the number of trials Similarly, for S ij , take the arithmetic mean of all trials (dim=0) to obtain a class activation map value corresponding to the spatial feature extraction layer independent of the number of trials For T' ij , take the arithmetic mean of the spatial dimension (dim=1) to obtain Superimpose T" ij for all test participants to obtain a class activation map value in the time domain that is independent of the number of participants and related to the class Map T" j according to the numerical value to a heat map, which is used as the result of the interpretability of the neural network feature time domain for each class. Corresponding to embodiments Figure 2 (B1) and (B2) background maps.

[0090] 4. Calculate the arithmetic mean of S' ij for all test participants in each class to obtain a value that is independent of the number of participants Select the time domain coordinate t j corresponding to the maximum value in T" j . As shown in embodiments Figure 2 (B1) and (B2), for the EEG signal with correct event feedback, i.e., j=1, t j = 0.2s, corresponding to the vertical dashed line in embodiment Figure 2 (B1) corresponding to the time coordinate. For the EEG signal with incorrect event feedback, i.e., j=0, t j = 0.365s, corresponding to the vertical dashed line in embodiment Figure 2 (B2) corresponding to the time coordinate. Select the corresponding value of S' j at t j Display s' j for all classes in the form of a brain topography map to obtain the interpretability result of the neural network feature in the spatial domain, corresponding to embodiments Figure 2 (C3) and (C4).

[0091] 5. Take the arithmetic mean of the significant trials (dim=0) for each test participant in the error-related negativity data set to obtain an EEG data Take the arithmetic mean of D ij for all test participants in each class to obtain a value that is independent of the number of participants Select the value corresponding to the Cz lead from D j ​​ d j That is, the amplitude of the EEG signal at different times. d is plotted based on the corresponding time. j The time-amplitude plot is the time-amplitude plot corresponding to lead Cz. Corresponding embodiment. Figure 2 The graphs in (B1) and (B2) are shown. For ease of comparison, the difference between the event-related potentials of the correct feedback event and the event-related potential of the incorrect feedback event in lead Cz is shown in the embodiment, and the time-amplitude curves are plotted (corresponding to...). Figure 2 (Dashline curves in (B1) and (B2)).

[0092] 6. Obtain D j In 4t j Brain topography information at different times This serves as a comparison of different types of EEG brain topography. (See example.) Figure 2 As shown in (C1) and (C2).

[0093] 7. Example Figure 2 From Figure 2 In (A), the solid lines drawn from significant trials represent analytical methods used in neuroscience. Figure 2 (A) The solid lines drawn from the convolutional layers represent the visualization results corresponding to the features learned by the neural network. (From the example) Figure 2 (B2) It can be seen that for the error feedback EEG signal (oddball event), the neural network features are mainly distributed around 0.365s in the time domain, which coincides with the maximum peak value of the difference between correct and error feedback ERPs in lead Cz. This proves that the classification features learned by the neural network are consistent with the prior assumptions of neuroscience. From the example Figure 2 (B1) It can be seen that, for the EEG signal of a correct feedback event, the distribution of the features learned by the neural network in the time domain is mainly concentrated around 0.2s. This time... Figure 2 (B2) The dashed lines show a clear difference in time, indicating that the features learned by the neural network have significant discriminative power across different categories. Furthermore, the neural network focuses on correct feedback events earlier than incorrect feedback events, which aligns with prior knowledge in neuroscience.

[0094] 8. Regarding the spatial characteristics of the P300-Speller signal, the EEG signal type studied in detail is the error feedback signal. Example Figure 2 (C4) reflects the brain topography maps corresponding to different categories of neural network features at 0.365s. From Figure 2As shown in (C4), for erroneous EEG signals, the spatial distribution of the features learned by the neural network is mainly concentrated in leads Cz, F7, P8, and O2. These lead regions are commonly used in neuroscience research on P300 signals. For correctly fed EEG signals, the features learned by the neural network at that time do not show a clear lead bias. This also indicates that at 0.365s, for different types of EEG signals, the features learned by the neural network not only have strong discriminative power in the time domain but also in the spatial domain (as shown in the example). Figure 2 (Compared to C3). From the example Figure 3 As can be seen in (C2), at 0.365s, there are also significant differences in the brain topography of correct feedback events and incorrect feedback events (as shown in the example). Figure 3 (Compared to C1). This indicates that the features learned by the neural network are consistent with prior knowledge in neuroscience, meaning that the features learned by the neural network from the error-correlated negative wave dataset are interpretable.

[0095] The interpretability of the features learned by the neural network for A03 and A06 in BCI4-2a is as follows:

[0096] 1. Using the BCI4-2a dataset, a participant-specific neural network model was trained. The classification results for each participant showed that A03 had the highest accuracy, while A06 had the lowest. Furthermore, regardless of whether neural network algorithms were used or manual feature extraction algorithms were employed, A03 consistently achieved high accuracy, while A06 consistently achieved low accuracy, also known as EEG-blind participants. These two participants were selected to examine the spatial distribution of the features learned by the neural network. The following description focuses on A03. The analysis method for A06 is the same as for A03.

[0097] 2. Using the trained neural network model, test the EEG data of A03, and obtain the top 10 trials with the highest confidence and correct predictions for each category. These are the significant trials corresponding to A03. Where j represents the category number, N represents the number of significant trials for each participant in each category, denoted as dim = 0; C represents the number of leads, denoted as dim = 1; and T represents the number of sampling points in each trial, denoted as dim = 2. In the BCI4-2a dataset, N = 10, C = 22, and T = 1125.

[0098] 3. Use class activation mapping, specifically Eigen-CAM technology, taking the salient values ​​of different classes A03 as input, to obtain the class activation mapping values ​​corresponding to the spatial feature extraction layer of the trained neural network model. Category activation mapping values ​​corresponding to the temporal feature extraction layer Where j represents the number of A03.

[0099] 4. To T j Take the arithmetic mean of the spatial dimension (dim = 1) to obtain a spatially independent, time-dependent neural network feature class activation mapping value Select T' j The first 10 maximum values of each trial, and record the corresponding time point, denoted as the set of significant times S t Use the set of significant times S t Make an index of S j in the time dimension (dim = 2) to obtain where T' = 10 represents the length of the set of significant times. Traverse S' j in each class (dim = 0) and make an index of S' j in each class of each trial (dim = 2) to obtain the mapping value of the neural network feature in the spatial dimension of each class of each trial Stack s' j according to the trial to obtain This is the visualization result of the neural network feature in space, which can be displayed in the form of a brain topography to obtain the attention degree of the neural network in classification to different leads. ​ The brain topography of the A03 and A06 neural network learned features in different classes is shown.

[0100] 5. From the results of the embodiment ​ It can be seen from the results that the brain topography of the neural network feature in A03 mainly focuses on C3 and C4 leads. Because the neural network is a discriminant model, it focuses on the decision boundary of different classes. This shows that C3 and C4 are the key to judging different motor imagery classes. This conclusion is consistent with the prior knowledge in neuroscience, and at the same time shows that the neural network learned features in A03 are reasonable. In different classes of A03, the distribution of the neural network learned features in space has obvious differences, which reflects the separability of the neural network learned features between different classes. At the same time, it also explains why A03 has a high classification accuracy. For A06, the neural network feature mainly focuses on the POz lead in space, and there is little difference between different classes. Looking back at the original EEG signal of A06, it is found that the signal amplitude of the POz lead is larger than that of other leads. This shows that the A06 participant was disturbed by noise when collecting the EEG data, resulting in the concentration of the neural network features on the POz lead, and the lower separability between different classes, thus having a lower classification accuracy. From the analysis results, the neural network feature visualization method proposed in the present application can display the motor imagery features learned by the neural network in the spatial dimension, and can provide an explanation of the neural network learned features that conforms to the prior knowledge in neuroscience.

[0101] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for interpretability of EEG features based on neural network decoding, characterized by the following steps: as follows: 1) The collected raw EEG data is preprocessed and then fed into the neural network for training; 2) Train the neural network and save the optimal neural network test model; 3) Using the trained neural network model, perform category prediction on the EEG data to be tested, label the EEG tests that the neural network correctly predicted, and perform interpretability analysis on the neural network model. When the EEG signal in step 3) is the event-related potential signal P300, the interpretability analysis is divided into neural network feature visualization, manual feature extraction visualization based on prior neuroscience knowledge, and comparative analysis of neural network feature visualization and manual feature extraction visualization. The specific steps are as follows: a) The steps for visualizing the features learned by the neural network are as follows: i. For the P300 signal, the features learned by the neural network are visualized using the EEG data of all participants. The top n trials with the highest prediction confidence for each participant and each category are called significant trials, denoted as [missing information]. subscript and These represent the participant's ID and the category ID, respectively. This represents the number of significant trials for each participant in each category, denoted as dim=0, indicating the 0th dimension; The number of leads is represented by dim=1; This represents the number of sampling points in each trial, denoted as dim=2; ii. Using Class Activation Mapping (CAM), with the salient trials of all classes in i as input, obtain the class activation map values ​​corresponding to the spatial feature extraction layer and the temporal feature extraction layer of the trained neural network model, denoted as […]. and ; iii. Targeting and Calculate the arithmetic mean of all trials, dim=0, to eliminate the influence of the number of trials on the class activation mapping value, denoted as . and ; iv. For each participant, for each category, Calculate the arithmetic mean of the spatial dimensions, dim=1, denoted as For all participants By overlaying the values, we obtain the time-domain category activation mapping value corresponding to each category, denoted as . ; v. to Based on the numerical values, a heatmap is mapped to serve as a visualization of the features learned by the neural network for each category in the time domain. vi. Calculate all participants to be tested in each category. The arithmetic mean, the result is denoted as This is to eliminate the influence of the number of participants on the visualization results of neural network features for each category, while recording... Time domain coordinates corresponding to the maximum value retrieval time The corresponding value is denoted as ; vii. To classify different categories The neural network features are visualized in the spatial domain by displaying them in the manner of brain topography. b) The steps for manually extracting and visualizing features based on prior neuroscience knowledge are as follows: i. Significant trials for each participant and each category in (a)-i Calculate the arithmetic mean of all trials where dim=0, resulting in an EEG dataset independent of the number of trials, denoted as . ; ii. Calculate all participants to be tested in each category. The arithmetic mean of the two is denoted as . ,from Select the value corresponding to lead Cz , That is, the amplitude of the EEG signal corresponding to lead Cz under different events, plotted. The time-amplitude curves were calculated, and the difference in amplitude between normal events and oddball events at the corresponding time in lead Cz was also calculated. iii. Choose from a)-vi Time coordinates corresponding to the maximum value Calculate and plot exist Brain topography at time ; c) Comparative analysis of neural network feature visualization and manually extracted feature visualization: i. Compare a)-v and b)-ii, and compare the neural network feature heatmap of the oddball event with the time-amplitude curve of lead Cz. If the time corresponding to the significant region in the neural network feature heatmap is related to the time corresponding to the relevant peak in b)-ii, and the relevant peak refers to the maximum positive peak of the oddball event or the maximum value of the difference between the normal event and the oddball event, then the localization of the features learned by the neural network in the time domain is considered reasonable. For the EEG signal of the normal event, if the above time is distinguishable from the time determined in the oddball event, and the neural network feature heatmap corresponds to some peaks in b)-ii, and some peaks include the peak of the normal event and the peak of the difference between the normal event and the oddball event, then the response of the features learned by the neural network to the normal event is considered reasonable. If the features learned by the neural network differ greatly from the above description, then the neural network cannot reasonably locate P300 in the time domain. Let ii be the time when the maximum value of the neural network feature heatmap occurs in the oddball event. That is, categories in a)-vi Let be the time corresponding to the oddball event; let be the time when the maximum value of the neural network feature heatmap occurs during normal events. Observe the differences between the brain topography maps of EEG signals in b)-iii and the brain topography maps of neural network features in a)-vii at these two time points; if Significant differences were observed in the EEG brain topography maps of different categories at times a)-vii and b)-iii, indicating that... Both time-based neural network features and neural network features exhibit good spatial distinguishability; the brain topography corresponding to the oddball event in a)-vii can further explain the importance of each lead in classification. In step 3), for event-related desynchronization (ERD) / event-related synchronization (ERS) signals represented by motion imagery signals, the interpretability analysis of neural network features provides spatial visualization of the features learned by the neural network. The specific steps are as follows: a) For motion imagery data, the interpretability of neural network features is based on the EEG data of a single participant. A trained neural network model is used to test the EEG data of a given participant, and the top n trials with the highest confidence and correct predictions for each category are denoted as significant trials. ; b) Using class activation mapping, with the salient trials of all classes in a) as input, obtain the class activation mapping values ​​corresponding to the spatial feature extraction layer of the trained neural network model. Category activation mapping values ​​corresponding to the temporal feature extraction layer ; c) Performing the arithmetic mean with spatial dimension dim=1 yields a spatially independent, temporally correlated neural network feature category activation mapping value, denoted as . ,choose The first m maximum values ​​of each trial are recorded, and their corresponding time points are denoted as the significance time set. ; d) Use the saliency time set in c) right By creating an index with time dimension dim=2, we obtain... , in Indicates the length of the significant time set; e) Iterate through each category with dim=0 trials. And for each category and each trial Summing by dim=2 in the time dimension yields the mapping value of the neural network features for each category and each trial in the spatial dimension. and will obtain Probabilistic approach; f) Superimposed according to the number of trials ,get The data is displayed in the form of brain topography, resulting in a spatial visualization of different categories of neural network features.

2. The method for interpretability of EEG features based on neural network decoding as described in claim 1, characterized in that, Step 1) The detailed preprocessing steps are as follows: a) Bandpass filtering b) Downsampling c) Trial segmentation d) Data normalization.

Images