Fast sequence visual presentation electroencephalogram classification method based on space-time progressive attention model

Abstract

The invention discloses a rapid sequence visual presentation electroencephalogram classification method based on a spatio-temporal progressive attention model. The method comprises the following steps: acquiring electroencephalogram data to be classified; inputting the to-be-classified electroencephalogram data into the trained time-space progressive attention model for processing to obtain the category of the to-be-classified electroencephalogram data; wherein the trained spatio-temporal progressive attention model is obtained by training the initial spatio-temporal progressive attention model by taking collected and processed preset data as a training data set and aiming at extracting spatial features and time features of the electroencephalogram data. The electroencephalogram data is obtained by collecting an image sequence watched by a testee in a rapid sequence visual presentation mode, and the image sequence comprises an infrared small target image and a non-target image. Useful information of electrodes and time slices can be utilized to the maximum extent, and meanwhile irrelevant information is reduced as much as possible.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model. Background Art

[0002] In recent years, significant progress has been made in brain-computer interface (BCI) technology, such as communication assistive devices and prosthetics. Among various BCI technologies, such as magnetoencephalography (MEG), electroencephalography (EEG), and functional magnetic resonance imaging (fMRI), EEG stands out for its non-invasiveness and portability, which is due to its ability to collect data by placing electrodes on the scalp. At the same time, EEG has high temporal resolution and cost-effectiveness, making EEG particularly suitable for various applications in BCI, including visual paradigms. Notably, EEG-based visual paradigms, such as steady-state visual evoked potentials (SSVEP) and rapid serial visual presentation (RSVP), have been shown to be very effective in exploiting the brain's strong response to visual stimuli. In the RSVP paradigm, a sequence of images is presented to the subject at a constant rate (usually 5-20 Hz), with the ratio of target images to non-target images being approximately 10% and 90%. These rare target images can induce significant event-related potential (ERP) activity, especially the P300 component in ERP, which is characterized by a high-amplitude response 250 to 500 milliseconds after target presentation. It can be used as a reliable indicator of target detection in BCI applications and has received continuous attention in recent years. In RSVP-based BCI systems, the key to improving system performance lies in how to effectively classify EEG signals, that is, to interpret the brain's activities in response to rapid sequence visual presentations.

[0003] To this end, researchers have introduced a variety of methods aimed at improving the decoding performance of EEG signals. For example, Sajda et al. introduced HDCA, pioneering a two-step method that combines Fisher linear discriminant analysis (FLD) for spatial weight training with logistic regression for temporal weight learning. Blankertz et al. introduced regularized linear discriminant analysis (rLDA) and further improved the classification of ERP signals, demonstrating the effectiveness of LDA with appropriate regularization by shrinkage. Li et al. proposed an ensemble learning method for EEG decoding in RSVP tasks, which uses an extreme gradient boosting framework to sequentially generate sub-models to improve consistency with the P300 pattern. Some researchers have tried to improve the EEG signal classification performance by increasing the number of electrodes, but found that an increase in the number of electrodes does not necessarily lead to better performance. Therefore, some studies have conducted a large number of experiments to find the best electrode combination, but found that the optimal combination varies from subject to subject, so it is impossible to select a uniform channel for all subjects. In recent years, with the adoption of deep learning methods, deep learning methods provide adaptive data representation learning and feature extraction, which significantly improves the classification performance of RSVP EEG data. Manor et al. introduced a new spatiotemporal regularization method through a three-layer convolutional neural network. In addition, given that the temporal resolution of EEG signals is much higher than the spatial resolution, more and more studies have begun to focus on the extraction of EEG temporal features. Lan found that there is information redundancy and interference noise in the time domain, and used a self-attention module to assign weights to each encoding segment. Li et al. proposed PPNN to prevent the phase information of EEG signals from being lost during the convolution process. Li et al. further considered the category imbalance problem of the RSVP task and proposed a deep reinforcement learning (DRL) model to alleviate its negative impact. However, these methods are not enough to fully explore the temporal dependencies in EEG signals.

[0004] In summary, the existing methods have the following main defects: 1. Most of the current EEG classification methods lack in-depth research on the EEG electrodes that contain more identification information. Current research has shown that the selection of different EEG electrodes in the RSVP task seriously affects the classification performance, and incorrect electrode selection may reduce the classification accuracy; 2. The existing technology is insufficient in how to use the temporal information of EEG signals. Traditional static EEG analysis methods may ignore valuable information. Excessive redundant EEG information may make EEG training more complicated and time-consuming, and may reduce the classification accuracy; 3. Most of the existing RSVP datasets are based on visible images of large targets. It is a challenge to study RSVP paradigms based on other types of images. Therefore, it is urgent to provide an EEG classification method to improve the above defects. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model, comprising:

[0007] Obtaining EEG data to be classified;

[0008] The EEG data to be classified are input into a trained spatiotemporal progressive attention model for processing to obtain the category of the EEG data to be classified; wherein, the trained spatiotemporal progressive attention model uses the preset data after collection and processing as a training data set, and is obtained after training the initial spatiotemporal progressive attention model for the purpose of extracting the spatial and temporal features of the EEG data. The preset data is obtained by collecting images of subjects viewing a sequence in a rapid sequence visual presentation manner, and the image sequence includes infrared small target images and non-target images.

[0009] Beneficial effects of the present invention:

[0010] The present invention provides a rapid serial visual presentation EEG classification method based on a spatiotemporal progressive attention model, which solves the challenges in existing methods by enhancing spatiotemporal connectivity and optimizing the use of rapid serial visual presentation (RSVP) related information, and enhances stability in classification tasks. This method enhances the processing and interpretation of EEG data by combining a graph-based structure and a progressive attention mechanism, so that the spatiotemporal progressive attention model can maximize the use of useful information from electrodes and time segments while minimizing irrelevant information to achieve better classification. In addition, we collected a small-target dim infrared image RSVP EEG dataset (named IRED), which will help expand the use scenarios of RSVP.

[0011] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model provided by an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram of a spatiotemporal progressive attention model provided by an embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of an infrared small target image provided by an embodiment of the present invention;

[0015] Figure 4is a schematic diagram of the overall RSVP paradigm provided by an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram of a progressive spatial learning module provided by an embodiment of the present invention;

[0017] Figure 6 It is a schematic diagram of a progressive time learning module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0019] See also Figure 1 , Figure 1 The present invention provides a flowchart of a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model. The present invention provides a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model, including:

[0020] S101, obtaining EEG data to be classified.

[0021] Specifically, in this embodiment, the EEG data to be classified is obtained by performing an EEG experiment after the subject wears an EEG cap.

[0022] S102, inputting the EEG data to be classified into the trained spatiotemporal progressive attention model for processing to obtain the category of the EEG data to be classified.

[0023] The trained spatiotemporal progressive attention model uses the collected and processed preset data as the training data set, and is obtained after the initial spatiotemporal progressive attention model is trained for the purpose of extracting the spatial and temporal features of the EEG data. The EEG data is obtained by collecting the subjects' viewing of image sequences in a rapid sequence visual presentation, and the image sequences include infrared small target images and non-target images.

[0024] Specifically, see Figure 2 , Figure 2 is a schematic diagram of a spatiotemporal progressive attention model provided by an embodiment of the present invention. In this embodiment, before training the initial spatiotemporal progressive attention model, it also includes:

[0025] A spatiotemporal progressive attention model is constructed, which includes a progressive spatial learning module, a convolutional layer, a first fully connected layer, a progressive temporal learning module, a pooling layer, a second fully connected layer and a softmax function. The output end of the progressive spatial learning module is connected to the input end of the convolutional layer, the output end of the convolutional layer is connected to the input end of the first fully connected layer, the output end of the first fully connected layer is connected to the input end of the progressive temporal learning module, the output end of the progressive temporal learning module is connected to the input end of the pooling layer, the output end of the pooling layer is connected to the input end of the second fully connected layer, and the softmax function is used to process the output result of the second fully connected layer; wherein,

[0026] The progressive spatial learning module includes a first spatial expert, a second spatial expert and a third spatial expert arranged in parallel. The first spatial expert, the second spatial expert and the third spatial expert have the same structure, all including a first graph convolutional network, a third fully connected layer, a first global average pooling layer and a first ReLU function; the progressive time learning module includes a progressive spatial learning channel. The progressive spatial learning channel includes a first time expert, a second time expert and a third time expert. The first time expert, the second time expert and the third time expert have the same structure, all including a second graph convolutional network, a fourth fully connected layer, a second global average pooling layer and a second ReLU function.

[0027] In this embodiment, the preset data collected and processed is used as a training data set, and the initial spatiotemporal progressive attention model is trained for the purpose of extracting the spatial and temporal features of the EEG data, including:

[0028] Acquire EEG data and construct a training data set; wherein the training data set includes multiple training samples;

[0029] Label the training samples in the training data set to obtain the true labels of the training samples;

[0030] The training data set and the true labels of the training samples in the training data set are input into the initial spatiotemporal progressive attention model. The spatial and temporal features of the EEG data are extracted, and iterative training is performed. The loss function value is minimized during a finite number of iterative training processes, and a trained spatiotemporal progressive attention model is obtained.

[0031] In this embodiment, obtaining EEG data and constructing a training data set include:

[0032] Acquire infrared small target images and non-target images, set the number of infrared small target images and non-target images in a ratio of 1:10 to 1:20, and construct an image sequence;

[0033] The subject wears an EEG cap and watches the image sequence in a rapid sequence visual presentation to obtain EEG data;

[0034] The EEG data is sampled and divided to obtain samples; the size of the samples is C×T, C represents the number of channels presented in the sample, each channel represents an electrode, and T represents the duration of the sample;

[0035] The samples are filtered and standardized to obtain processed samples, which are used as training samples to construct a training data set.

[0036] Specifically, in this embodiment, data is obtained through RSVP experiment, that is, 8 subjects (5 males and 3 females, aged between 20-30 years old) wear EEG caps and watch the image sequence in a rapid sequence visual presentation mode to obtain the EEG data of the subjects; the subjects have normal or corrected-to-normal vision and no history of mental illness. The experiment follows the principles of the Declaration of Helsinki. All subjects understand the experimental process and agree by signing the consent form. This experiment uses infrared images from the LSOTB-TIR dataset. First, select images with small infrared targets and resize them to 1280×720 pixels to obtain 900 small infrared target images. Please refer to Figure 3 , Figure 3 is a schematic diagram of an infrared small target image provided by an embodiment of the present invention, wherein the target includes an airplane, an animal, a helicopter, a car, and a person; see Figure 4 , Figure 4 It is a schematic diagram of the overall RSVP paradigm provided by an embodiment of the present invention. The text of the target type of the current experiment is displayed on the screen, and the fixation is displayed for 1000 milliseconds to ensure that the subject's attention is focused on the center of the screen. In the experiment, infrared small target images and non-target images are displayed in sequence. In each experiment, 45 images are displayed at a rate of 5 Hz, and there are 2 to 4 infrared small target images in the 45 images, and the infrared small target images appear randomly; the whole experiment includes 18 blocks, and each block contains 12 experiments. It should be noted that in each block, except for the image prompted as the target, the other four types of images will be used as the background image of the subject. Between every two experiments, the subject is advised to take a break. In this embodiment, the RSVP experimental paradigm code is implemented by Psychopy.

[0037] The EEG data generated by the subjects during the experiment were recorded using the BioSemi-ActiveTwo system, which is based on the standard 10-20 system. The EEG cap has 64 scalp electrodes, a sampling rate of 1024Hz, and the impedance of each electrode is kept below 20kΩ to ensure high-quality signals. The collected EEG data were preprocessed as follows:

[0038] 1. Sample segmentation. For the collected EEG data, it is first segmented into EEG data samples, and the samples start from 1 second after each stimulus. Each subject provided approximately 1280 EEG samples. For each sample, its size is C×T, where C represents the number of electrodes presented in the sample and T represents the sample duration.

[0039] 2. Filtering and normalization: After sample segmentation, all samples are sent to a 6th-order Butterworth bandpass filter with a cutoff frequency of 1 to 40 Hz, and then normalized using the z-score method.

[0040] The selected public dataset contains RSVP EEG data from 8 healthy volunteers (7 females and 1 male, aged 19-29 years old). The volunteers were asked to wear the non-invasive EEG acquisition device of the BioSemi-ActiveTwo system, which was equipped with 256 scalp electrodes and a sampling frequency of 256Hz. During the data collection process, the volunteers were required to sit in front of a CRT screen in a dim environment for 256 trackings. Each trial was presented with a sequence of 50 images at a frequency of 12Hz, two of which were target images containing "airplanes". The EEG data of each track was divided into 1-second slices as training samples, and a 256×256 matrix was obtained. These samples were pre-processed with data such as bandpass filtering and a cutoff frequency of 0.1-48Hz.

[0041] In this experiment, classification accuracy (acc) and standard deviation (std) are used as criteria for comparing the performance of different methods. Specifically, for the collected dataset, it contains EEG data of 8 healthy subjects, and each subject has approximately 1280 samples, including target and non-target samples. In order to divide the training dataset and the test dataset, 75% of the EEG data of each subject was selected as training data, while the remaining 25% was designated as test data. For the public dataset, the data of the remaining six subjects were used in subsequent experiments because one subject only provided training data and the data of another subject was corrupted. In the public dataset, the division of the training dataset and the test dataset is predefined, and the final accuracy is obtained by calculating the average accuracy of each subject.

[0042] In this embodiment, the training data set and the true labels of the training samples in the training data set are input into the initial spatiotemporal progressive attention model, and iterative training is performed for the purpose of extracting the spatial features and temporal features of the EEG data, including:

[0043] For the nth iteration training process, the training data set is input into the progressive spatial learning module to extract the spatial features of the EEG data and obtain the spatial feature vector of the EEG data;

[0044] Input the EEG data spatial feature vector into the convolution layer and the first fully connected layer for processing to obtain a low-dimensional EEG data spatial feature vector;

[0045] Inputting the low-dimensional EEG data spatial feature vector into the progressive time learning module, extracting the EEG data temporal features, and obtaining the EEG data temporal feature vector;

[0046] Input the temporal feature vector of the EEG data into the pooling layer for processing to obtain the spatial and temporal feature vectors of the EEG data;

[0047] The spatial and temporal feature vectors of EEG data are input into the second fully connected layer and mapped to the category space by the softmax function to obtain the predicted category.

[0048] In this embodiment, see Figure 5 , Figure 5 is a schematic diagram of a progressive spatial learning module provided by an embodiment of the present invention. A training data set is input into the progressive spatial learning module, and the spatial features of the EEG data are extracted to obtain a spatial feature vector of the EEG data, including:

[0049] The training samples are cut into multiple slices according to the time series, and the i-th slice is represented as:

[0050]

[0051] Among them, x i,C represents the data of the Cth channel (electrode) of the i-th slice;

[0052] According to the i-th slice of the training sample, the first spatial undirected graph is constructed, and its expression is:

[0053]

[0054] in, represents the set of nodes of the spatial undirected graph, ε represents the set of edges of the spatial undirected graph, An adjacency matrix representing the spatial relationships between nodes of a spatial undirected graph according to electrode locations;

[0055] The first spatial undirected graph is input into the first graph convolutional network of the first spatial expert in the progressive learning channel for processing to obtain the first spatial feature vector H i , whose expression is:

[0056]

[0057] Among them, δ(·) represents the activation function, Represents the parameters to be learned in the graph convolutional network, Lap S represents the normalized Laplacian matrix, Tk (·) represents the k-order Chebyshev polynomial, d represents the dimension of the first spatial eigenvector, h i,Cd represents the feature value of the Cth row and dth column in the feature obtained by the i-th expert;

[0058] According to the first spatial eigenvector H i , the probability distribution predicted by the spatial expert is generated through the third fully connected layer, and the probability distribution predicted by the spatial expert is compared with the true label of the training sample to obtain the first loss function;

[0059] Calculate the first spatial eigenvector H i The partial linearization slope of the j∈{1,2,...,d}th column feature and category z in is used to calculate the importance weight of the jth column feature through the first global average pooling layer according to the partial linearization slope. Its expression is:

[0060]

[0061] Where C represents the total number of channels, h i,cj represents the feature value of the Cth row and jth column in the feature obtained by the i-th expert, c represents the channel (i.e., electrode), Indicates partial derivative, f1 S (X i ) represents the probability distribution of the prediction of the i-th slice by the first expert, [·] z represents the probability of the zth class;

[0062] The first spatial eigenvector H i The importance weights of all column features in are multiplied by their corresponding column features and then summed to obtain the heat value of the electrode corresponding to the c-th row feature. The electrodes with negative effects are excluded by the first ReLU function, and its expression is:

[0063]

[0064] Get the first spatial eigenvector H i The thermal values ​​of the electrodes corresponding to all row features in the training sample are used to construct the first spatial attention map of the i-th slice. The first spatial attention map The value in is compared with the first threshold ε, and its expression is:

[0065]

[0066] The electrodes corresponding to the values ​​greater than the first threshold ε are taken as important electrodes, and the first important electrode set is constructed.

[0067]

[0068] The first important electrode set is used to prune the first space undirected graph to obtain the second space undirected graph. The second space expert processes the second space undirected graph to obtain the second space feature vector H' i , and get the second spatial attention map and the second important electrode set X' i '; According to the second important electrode set X' i 'Prune the undirected graph in the second space to obtain the undirected graph in the third space. The third space expert processes the undirected graph in the third space to obtain the third space vector H' i ', the first spatial eigenvector H i , the second space eigenvector H' i and the third space vector H' i 'Merge, as the spatial feature vector output by the progressive learning module, the expression is:

[0069]

[0070] The M slices of the training sample are processed by the progressive learning module to output M spatial feature vectors, and the spatial feature vector of the EEG data is obtained according to the M spatial feature vectors.

[0071] In this embodiment, the first loss function L S The expression is:

[0072]

[0073] Among them, X(n) represents the number of slices of the nth training sample, represents the true label of the nth training sample, f1 S (X i (n)) represents the probability distribution of the spatial expert’s prediction for the ith slice of the nth training sample, [·] z represents the probability of the zth class, M represents the total number of training sample slices, and N represents the total number of training samples.

[0074] In this embodiment, the EEG data spatial feature vector is input into the convolution layer and the first fully connected layer for processing to obtain a low-dimensional EEG data spatial feature vector, including:

[0075] Convolution and full connection operations are performed on the EEG data spatial feature vector to reduce the dimension and obtain low-dimensional spatial features. Sort by time to get the low-dimensional EEG data spatial feature vector

[0076]

[0077] In this embodiment, see Figure 6 , Figure 6 is a schematic diagram of a progressive time learning module provided by an embodiment of the present invention, inputting a low-dimensional EEG data spatial feature vector into the progressive time learning module, extracting the EEG data time feature, and obtaining the EEG data time feature vector, including:

[0078] According to the low-dimensional EEG data spatial feature vector Constructing the first time undirected graph Its expression is:

[0079]

[0080] Among them, A T It is an adjacency matrix that represents the temporal relationship between nodes of a temporal undirected graph according to the temporal position of the spatial features of EEG data;

[0081] The first-time undirected graph is input into the second graph convolutional network of the first-time expert in the progressive time learning module for processing to obtain the first-time feature vector O, which is expressed as:

[0082]

[0083] Among them, δ(·) represents the activation function, Represents the parameters to be learned in the graph convolutional network, Lap T represents the normalized Laplacian matrix, T k (·) represents the k-th order Chebyshev polynomial, d″ represents the dimension of the first time eigenvector;

[0084] According to the first time feature vector O, a probability distribution predicted by the time expert is generated through the fourth fully connected layer, and the probability distribution predicted by the time expert is compared with the true label of the training sample to obtain a second loss function;

[0085] Through the second global average pooling layer, the weight of the feature of the first time feature vector O is calculated. Multiply the importance weights of all column features in the first time feature vector O by their corresponding column features and then sum them up to get the heat value of the time segment corresponding to the Mth row feature

[0086] Obtain the thermal values ​​of the electrodes corresponding to all row features in the first time feature vector O, and construct the first time attention map of the training sample The first time attention map Φ T The value in is compared with the second threshold ω, and the time segments corresponding to the values ​​greater than the second threshold are taken as important segments to construct a first important time segment set;

[0087] The first time undirected graph is pruned according to the first important time segment set to obtain the second time undirected graph. The second time expert processes the second time undirected graph to obtain the second time feature vector O' and the second time attention graph Φ T ' and a second important time segment set; the second time undirected graph is pruned according to the second important time segment to obtain a third time undirected graph, and the third time expert processes the third time undirected graph to obtain a third time feature vector O", and the first time feature vector O, the second time feature vector O' and the third time feature vector O" are merged as the time feature vector output by the progressive time learning module, that is, the time feature vector of the EEG data.

[0088] In this embodiment, the expression of the second loss function is:

[0089]

[0090] in, Represents the true label of the nth training sample.

[0091] In this embodiment, the EEG data temporal feature vector is input into the pooling layer for processing to obtain the EEG data spatial and temporal feature vectors, which are expressed as follows:

[0092]

[0093] in, Contains spatial and temporal information.

[0094] In this embodiment, the spatial and temporal feature vectors of the EEG data are input into the second fully connected layer and mapped to the category space by the softmax function to obtain the predicted category, including:

[0095] The spatial and temporal feature vectors of EEG data Flatten to get the flattened vector

[0096] The flattened vector is input into the second fully connected layer and the softmax function for processing to obtain the logistic regression layer vector of the probability distribution, which is expressed as:

[0097]

[0098] Among them, Z represents the number of classes, W represents the weight, and b represents the bias;

[0099] The expression of the probability P of the training sample X being classified into the zth class is:

[0100]

[0101] Among them, ei represents the value of the i-th category in e;

[0102] The category corresponding to the highest probability is taken as the predicted category, and its expression is:

[0103]

[0104] in, Represents the predicted category.

[0105] In summary, the present invention provides a rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model, which has the following beneficial effects:

[0106] 1. Capable of extracting the most important electrodes; the present invention adopts a progressive spatial learning (PSL) module to extract EEG data features from more relevant electrodes. To achieve this goal, three spatial experts are constructed. The previous spatial expert selects important electrodes and inputs them into the next spatial expert. The process continues to finely filter out the electrodes most relevant to the EEG data for EEG data classification in the RSVP task.

[0107] 2. Ability to extract important time segments: After obtaining the spatial features of all EEG data, the present invention uses a progressive temporal learning (PTL) module to gradually select important EEG time segments and generate spatiotemporal EEG features for final classification. The present invention also uses progressive attention to selectively focus on key time segments through training sample slices, thereby minimizing the impact of the time difference of related potentials on classification.

[0108] 3. Construct a specific RSVP paradigm; In order to solve the shortcomings of RSVP task application based on visible images, the present invention constructs a new dataset IRED, which consists of EEG data of 8 subjects, and the EEG data are induced by weak infrared images with small targets; IRED is very helpful for exploring the application of RSVP paradigm in low-light or nighttime environments, and enriches the diversity of available stimuli for RSVP research.

[0109] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant is intended to cover non-exclusive inclusion, so that the article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the article or device including the elements. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The orientation or position relationship indicated by "up", "down", "left", "right", etc. is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0111] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A rapid sequence visual presentation EEG classification method based on a spatiotemporal progressive attention model, characterized in that: include: Obtaining EEG data to be classified; Inputting the EEG data to be classified into a trained spatiotemporal progressive attention model for processing to obtain the category of the EEG data to be classified; Among them, the trained spatiotemporal progressive attention model uses the collected and processed preset data as the training data set, and is obtained after training the initial spatiotemporal progressive attention model for the purpose of extracting the spatial and temporal characteristics of the EEG data. The preset data is obtained by collecting the subjects' viewing of an image sequence in a rapid serial visual presentation manner, and the image sequence includes infrared small target images and non-target images.

2. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 1 is characterized in that: Before training the initial spatiotemporal progressive attention model, it also includes: Constructing a spatiotemporal progressive attention model, the spatiotemporal progressive attention model includes a progressive spatial learning module, a convolutional layer, a first fully connected layer, a progressive temporal learning module, a pooling layer, a second fully connected layer and a softmax function, wherein the output end of the progressive spatial learning module is connected to the input end of the convolutional layer, the output end of the convolutional layer is connected to the input end of the first fully connected layer, the output end of the first fully connected layer is connected to the input end of the progressive temporal learning module, the output end of the progressive temporal learning module is connected to the input end of the pooling layer, the output end of the pooling layer is connected to the input end of the second fully connected layer, and the softmax function is used to process the output result of the second fully connected layer; wherein, The progressive spatial learning module includes a first spatial expert, a second spatial expert and a third spatial expert. The first spatial expert, the second spatial expert and the third spatial expert have the same structure, and all include a first graph convolutional network, a third fully connected layer, a first global average pooling layer and a first ReLU function; the progressive temporal learning module includes a first temporal expert, a second temporal expert and a third temporal expert. The first temporal expert, the second temporal expert and the third temporal expert have the same structure, and all include a second graph convolutional network, a fourth fully connected layer, a second global average pooling layer and a second ReLU function.

3. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 2 is characterized in that: The collected and processed preset data is used as the training data set, and the initial spatiotemporal progressive attention model is trained for the purpose of extracting the spatial and temporal features of the EEG data, including: Acquire EEG data and construct a training data set; wherein the training data set includes a plurality of training samples; Labeling the training samples in the training data set to obtain true labels of the training samples; The training data set and the true labels of the training samples in the training data set are input into the initial spatiotemporal progressive attention model, and iterative training is performed for the purpose of extracting the spatial and temporal features of the EEG data. The loss function value is minimized during a finite number of iterative training processes, thereby obtaining the trained spatiotemporal progressive attention model.

4. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 3 is characterized in that: Obtain EEG data and construct a training data set, including: Acquire infrared small target images and non-target images, set the number of infrared small target images and non-target images in a ratio of 1:10 to 1:20, and construct an image sequence; The subject wears an EEG cap and watches the image sequence in a rapid sequence visual presentation manner to obtain the subject's EEG data; Performing sample segmentation on the EEG data to obtain samples; wherein the size of the samples is C×T, C represents the number of channels presented in the sample, and T represents the duration of the sample; The samples are filtered and standardized to obtain processed samples, which are used as training samples to construct a training data set.

5. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 3 is characterized in that: Inputting the training data set and the true labels of the training samples in the training data set into the initial spatiotemporal progressive attention model, and performing iterative training for the purpose of extracting the spatial features and temporal features of the EEG data, including: For the nth iteration training process, the training data set is input into the progressive spatial learning module, the spatial features of the EEG data are extracted, and the spatial feature vector of the EEG data is obtained; Inputting the EEG data spatial feature vector into the convolutional layer and the first fully connected layer for processing to obtain a low-dimensional EEG data spatial feature vector; Inputting the low-dimensional EEG data spatial feature vector into the progressive time learning module, extracting the EEG data time feature, and obtaining the EEG data time feature vector; Inputting the temporal feature vector of the EEG data into the pooling layer for processing to obtain spatial and temporal feature vectors of the EEG data; The spatial and temporal feature vectors of the EEG data are input into the second fully connected layer and mapped to the category space by the softmax function to obtain the predicted category.

6. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 5 is characterized in that: Inputting the training data set into the progressive spatial learning module, extracting the spatial features of the EEG data, and obtaining the spatial feature vector of the EEG data, including: The training samples are cut into multiple slices according to the time series, and the i-th slice is represented as: Among them, x i,C Represents the data of the Cth channel of the i-th slice; According to the i-th slice of the training sample, a first spatial undirected graph is constructed, and its expression is: in, represents the set of nodes of the spatial undirected graph, ε represents the set of edges of the spatial undirected graph, An adjacency matrix representing the spatial relationships between nodes of a spatial undirected graph according to electrode locations; The first spatial undirected graph is input into the first graph convolutional network of the first spatial expert in the progressive spatial learning module for processing to obtain a first spatial feature vector H i , whose expression is: Among them, δ(·) represents the activation function, Represents the parameters to be learned in the graph convolutional network, Lap S represents the normalized Laplacian matrix, T k (·) represents the k-order Chebyshev polynomial, d represents the dimension of the spatial eigenvector, h i,Cd represents the feature value of the Cth row and dth column in the feature obtained by the i-th spatial expert; According to the first spatial eigenvector H i , generating a probability distribution predicted by a spatial expert through a third fully connected layer, and comparing the probability distribution predicted by the spatial expert with the true label of the training sample to obtain a first loss function; Calculate the first spatial eigenvector H i The partial linearization slope of the j∈{1,2,...,d}th column feature and category z in , and the importance weight of the jth column feature is calculated through the first global average pooling layer according to the partial linearization slope Its expression is: Where C represents the total number of channels presented in the slice, and h i,cj represents the feature value of the Cth row and jth column in the feature obtained by the i-th spatial expert, where c represents the channel, represents partial derivative, represents the probability distribution predicted by the spatial expert for the i-th slice, [·] z represents the probability of the zth class; The first spatial eigenvector H i The importance weights of all column features in are multiplied by their corresponding column features and then summed to obtain the heat value of the electrode corresponding to the Cth row feature. The electrodes with negative effects are excluded by the first ReLU function, and its expression is: Get the first spatial feature vector H i The thermal values ​​of the electrodes corresponding to all row features in the training sample are used to construct the first spatial attention map of the i-th slice of the training sample. The first spatial attention map The value in is compared with the first threshold ε, and its expression is: The electrodes corresponding to the values ​​greater than the first threshold ε are taken as important electrodes to construct the first important electrode set. The first important electrode set is used to prune the first spatial undirected graph to obtain a second spatial undirected graph. The second spatial expert processes the second spatial undirected graph to obtain a second spatial feature vector H' i , and get the second spatial attention map and the second important electrode set X' i '; According to the second important electrode set X' i 'Prune the second space undirected graph to obtain a third space undirected graph, and the third space expert processes the third space undirected graph to obtain a third space vector H' i ', the first spatial eigenvector H i , the second spatial eigenvector H' i and the third space vector H' i 'Merge, as the spatial feature vector output by the progressive spatial learning module; The M slices of the training sample are processed by a plurality of the progressive spatial learning modules to output M spatial feature vectors, and the spatial feature vectors of the EEG data are obtained according to the M spatial feature vectors.

7. The rapid sequence visual presentation EEG classification method based on the spatiotemporal progressive attention model according to claim 6 is characterized in that: The first loss function L S The expression is: Among them, X(n) represents the number of slices of the nth training sample, represents the true label of the nth training sample, represents the probability distribution of the spatial expert's prediction for the i-th slice of the n-th training sample, [·] z represents the probability of the zth class, M represents the total number of training sample slices, and N represents the total number of training samples.

8. The method for rapid sequence visual presentation EEG classification based on spatiotemporal progressive attention model according to claim 5, characterized in that: Inputting the low-dimensional EEG data spatial feature vector into the progressive time learning module, extracting the EEG data time feature, and obtaining the EEG data time feature vector, including: According to the low-dimensional EEG data spatial feature vector Constructing the first time undirected graph Its expression is: Among them, A T An adjacency matrix representing the temporal relationship between nodes of a temporal undirected graph according to the temporal positions of the spatial features of the EEG data; The first time undirected graph is input into the second graph convolutional network of the first time expert in the progressive time learning module for processing to obtain a first time feature vector O, which is expressed as: Among them, δ(·) represents the activation function, Represents the parameters to be learned in the graph convolutional network, Lap T represents the normalized Laplacian matrix, T k (·) represents the k-th order Chebyshev polynomial, d″ represents the dimension of the first time eigenvector; According to the first time feature vector O, a probability distribution predicted by a time expert is generated through a fourth fully connected layer, and the probability distribution predicted by the time expert is compared with the true label of the training sample to obtain a second loss function; Through the second global average pooling layer, the weight of the feature of the j∈{1,2,...,d″}th column in the first time feature vector O is calculated The importance weights of all column features in the first time feature vector O are multiplied by their corresponding column features and then summed to obtain the heat value of the time segment corresponding to the Mth row feature. Obtain the thermal values ​​of the electrodes corresponding to all row features in the first time feature vector O, and construct the first time attention map of the training sample The first time attention map Φ T The value in is compared with the second threshold ω, and the time segments corresponding to the values ​​greater than the second threshold are taken as important segments to construct a first important time segment set; The first time undirected graph is pruned according to the first important time segment set to obtain a second time undirected graph, and the second time expert processes the second time undirected graph to obtain a second time feature vector O' and a second time attention graph Φ' T and a second important time segment set; pruning the second time undirected graph according to the two important time segment sets to obtain a third time undirected graph, the third time expert processes the third time undirected graph to obtain a third time feature vector O", merging the first time feature vector O, the second time feature vector O' and the third time feature vector O" as the output feature vector of the progressive time learning module.

9. The method for rapid sequence visual presentation EEG classification based on spatiotemporal progressive attention model according to claim 8, characterized in that: The expression of the second loss function is: in, Represents the true label of the nth training sample.

10. The rapid sequence visual presentation EEG classification method based on spatiotemporal progressive attention model according to claim 5, characterized in that: The spatial and temporal feature vectors of the EEG data are input into the second fully connected layer and mapped to the category space by the softmax function to obtain the predicted category, including: The spatial and temporal feature vectors of the EEG data Flatten to get the flattened vector The flattened vector is input into the second fully connected layer and the softmax function for processing to obtain a logistic regression layer vector of probability distribution, which is expressed as: Among them, Z represents the number of classes, W represents the weight value, and b represents the bias; The expression of the probability P of the training sample X being classified into the zth class is: Among them, e i represents the value of the i-th category in e; The category corresponding to the highest probability is taken as the predicted category, and its expression is: in, Represents the predicted category.

Images