Rsvp eeg classification method based on spatiotemporal progressive attention model

Abstract

The application discloses a kind of based on fast sequence visual presentation electroencephalogram classification method of spatiotemporal progressive attention model, comprising: obtaining the electroencephalogram data to be classified;The electroencephalogram data to be classified is input to the trained spatiotemporal progressive attention model and is processed, obtains the category of electroencephalogram data to be classified;Wherein, the trained spatiotemporal progressive attention model with the preset data after acquisition and processing as training data set, to extract the spatial feature and time feature of electroencephalogram data as the purpose, after training to initial spatiotemporal progressive attention model, obtain.The electroencephalogram data is obtained by collecting subject in the mode of fast sequence visual presentation and watching image sequence, and the image sequence includes infrared small target image and non-target image.The present application can maximize the use of electrode and time segment useful information, while minimizing irrelevant information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a RSVP electroencephalogram classification method based on a space-time progressive attention model. BACKGROUND

[0002] In recent years, brain-computer interface (BCI) technology has made significant progress, such as communication auxiliary devices and prostheses. Among various BCI technologies, such as magnetoencephalography (MEG), electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), EEG stands out due to its non-invasiveness and portability, which is attributed to the fact that EEG can collect data by placing electrodes on the scalp, while EEG has high temporal resolution and cost-effectiveness, making it particularly suitable for various applications in BCI, including visual paradigms. Notably, EEG-based visual paradigms, such as steady-state visual evoked potentials (SSVEP) and rapid serial visual presentation (RSVP), have been proven to be very effective in utilizing the brain's strong response to visual stimuli. In the RSVP paradigm, a sequence of images is presented to the subject at a constant rate (usually 5-20 Hz), with the ratio of target images and non-target images being approximately 10% and 90%. These rare target images can induce significant event-related potential (ERP) activity, particularly the P300 component in ERP, which is characterized by a high-amplitude response occurring 250-500 milliseconds after target presentation, and can serve as a reliable indicator for target detection in BCI applications, which has received continuous attention in recent years. In RSVP-based BCI systems, the key to improving system performance lies in how to effectively classify EEG signals, i.e., to interpret the brain's activity generated by rapid serial visual presentation.

[0003] To this end, researchers have introduced a variety of methods aimed at improving the decoding performance of EEG signals. For example, Sajda et al. introduced HDCA, which pioneered a two-step method combining Fisher linear discriminant analysis (FLD) for spatial weight training and logistic regression for temporal weight learning. Blankertz et al. introduced regularized linear discriminant analysis (rLDA) and further improved the classification of ERP signals, demonstrating the effectiveness of LDA with appropriate regularization by shrinkage. Li et al. proposed an ensemble learning method for EEG decoding in RSVP tasks, which uses an extreme gradient boosting framework to sequentially generate sub-models, thereby improving consistency with P300 patterns. Some researchers have attempted to improve EEG signal classification performance by increasing the number of electrodes, but found that increasing the number of electrodes does not necessarily result in better performance. Therefore, some studies have conducted extensive experiments to find the optimal electrode combination, but found that the optimal combination varies from subject to subject, making it impossible to select a uniform channel for all subjects. In recent years, with the adoption of deep learning methods, deep learning methods provide adaptive data representation learning and feature extraction, significantly improving the classification performance of RSVP EEG data. Manor et al. introduced a new spatiotemporal regularization method through a three-layer convolutional neural network. In addition, given the much higher temporal resolution than spatial resolution of EEG signals, more and more research has begun to focus on the extraction of EEG temporal features. Lan found that there is information redundancy and interference noise in the time domain, and used a self-attention module to assign weights to each encoding segment. Li et al. proposed PPNN to prevent the loss of phase information of EEG signals during convolution. Li et al. further considered the class imbalance problem of RSVP tasks and proposed a deep reinforcement learning (DRL) model to mitigate its negative effects. However, these methods are not sufficient to fully explore the temporal dependencies in EEG signals.

[0004] In summary, the existing methods mainly have the following defects: 1. Most of the current EEG classification methods lack in-depth research on the discriminative information contained in the EEG electrodes. Current research has shown that selecting different EEG electrodes in RSVP tasks severely affects the classification performance, and incorrect electrode selection can reduce the classification accuracy; 2. The existing technology is insufficient in terms of how to utilize the temporal information of EEG signals. Valuable information that can be ignored by traditional static EEG analysis methods can make EEG training more complex and time-consuming, and can potentially reduce classification accuracy; 3. Most of the existing RSVP datasets are based on visible images of large targets, and it is a challenge to study RSVP paradigms based on other types of images. Therefore, there is an urgent need to provide an EEG classification method to improve the above-mentioned defects. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the application provides a fast sequence visual presentation electroencephalogram classification method based on a space-time progressive attention model.

[0006] In a first aspect, the application provides a fast sequence visual presentation electroencephalogram classification method based on a space-time progressive attention model, comprising:

[0007] Obtaining electroencephalogram data to be classified;

[0008] Inputting the electroencephalogram data to be classified into a trained space-time progressive attention model for processing to obtain the category of the electroencephalogram data to be classified; wherein the trained space-time progressive attention model takes the pre-set data collected and processed as the training data set, is trained for the purpose of extracting the spatial and temporal features of the electroencephalogram data, and is obtained after the initial space-time progressive attention model is trained, the pre-set data is obtained by collecting the subjects watching the image sequence in the form of fast sequence visual presentation, and the image sequence includes infrared small target images and non-target images.

[0009] The application has the following beneficial effects:

[0010] The fast sequence visual presentation electroencephalogram classification method based on the space-time progressive attention model provided by the application solves the challenges in the prior art by enhancing the space-time connectivity and optimizing the use of fast sequence visual presentation (RSVP) related information, thereby enhancing the stability in the classification task. The method combines the graph-based structure and the progressive attention mechanism to enhance the processing and interpretation of the electroencephalogram data, so that the space-time progressive attention model can maximize the use of useful information of the electrodes and time segments and minimize the irrelevant information as much as possible to achieve better classification. In addition, we collect the dim infrared image RSVP EEG dataset based on small targets (named IRED), which will help to expand the use scenarios of RSVP.

[0011] The application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flowchart of the fast sequence visual presentation electroencephalogram classification method based on the space-time progressive attention model provided by the embodiments of the application;

[0013] Figure 2 is a schematic diagram of the space-time progressive attention model provided by the embodiments of the application;

[0014] Figure 3 is a schematic diagram of the infrared small target image provided by the embodiments of the application;

[0015] Figure 4is a schematic diagram of the overall RSVP paradigm provided by the embodiment of the present application;

[0016] Figure 5 is a schematic diagram of the progressive spatial learning module provided by the embodiment of the present application;

[0017] Figure 6 is a schematic diagram of the progressive temporal learning module provided by the embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0019] Please refer to Figure 1 , Figure 1 is a flowchart of the fast sequence visual presentation electroencephalogram classification method based on the spatiotemporal progressive attention model provided by the embodiment of the present application. The fast sequence visual presentation electroencephalogram classification method based on the spatiotemporal progressive attention model provided by the present application comprises:

[0020] S101, acquiring the electroencephalogram data to be classified.

[0021] Specifically, in the present embodiment, the electroencephalogram data to be classified is acquired through an electroencephalogram experiment after the subject to be tested wears an electroencephalogram cap.

[0022] S102, inputting the electroencephalogram data to be classified into the trained spatiotemporal progressive attention model for processing to obtain the category of the electroencephalogram data to be classified.

[0023] The trained spatiotemporal progressive attention model takes the pre-set data collected and processed as the training data set, and is obtained by training the initial spatiotemporal progressive attention model for the purpose of extracting the spatial features and temporal features of the electroencephalogram data. The electroencephalogram data is obtained by collecting the subject's viewing of the image sequence in the form of fast sequence visual presentation. The image sequence includes infrared small target images and non-target images.

[0024] Specifically, please refer to Figure 2 , Figure 2 is a schematic diagram of the spatiotemporal progressive attention model provided by the embodiment of the present application. In the present embodiment, before training the initial spatiotemporal progressive attention model, it further comprises:

[0025] The spatio-temporal progressive attention model comprises a progressive spatial learning module, a convolutional layer, a first full connection layer, a progressive temporal learning module, a pooling layer, a second full connection layer and a softmax function, an output end of the progressive spatial learning module is connected with an input end of the convolutional layer, an output end of the convolutional layer is connected with an input end of the first full connection layer, an output end of the first full connection layer is connected with an input end of the progressive temporal learning module, an output end of the progressive temporal learning module is connected with an input end of the pooling layer, an output end of the pooling layer is connected with an input end of the second full connection layer, and the softmax function is used for processing an output result of the second full connection layer; wherein,

[0026] The progressive spatial learning module comprises the first, second and third spatial experts arranged in parallel, the first, second and third spatial experts have the same structure and each comprises a first graph convolutional network, a third full connection layer, a first global average pooling layer and a first ReLU function; the progressive temporal learning module comprises a progressive spatial learning channel, the progressive spatial learning channel comprises the first, second and third temporal experts, the first, second and third temporal experts have the same structure and each comprises a second graph convolutional network, a fourth full connection layer, a second global average pooling layer and a second ReLU function.

[0027] In the embodiment, the preset data after being collected and processed is taken as a training data set, and the initial spatio-temporal progressive attention model is trained for the purpose of extracting spatial features and temporal features of the electroencephalogram data, including:

[0028] The electroencephalogram data is acquired, and a training data set is constructed; wherein the training data set comprises a plurality of training samples;

[0029] The training samples in the training data set are labeled to obtain real labels of the training samples;

[0030] The training data set and the real labels of the training samples in the training data set are input into the initial spatio-temporal progressive attention model, and the initial spatio-temporal progressive attention model is iteratively trained for the purpose of extracting spatial features and temporal features of the electroencephalogram data, and the loss function value reaches the minimum in the limited iteration training process, and a trained spatio-temporal progressive attention model is obtained.

[0031] In the embodiment, the electroencephalogram data is acquired, and a training data set is constructed, including:

[0032] The infrared small target image and the non-target image are acquired, and the number of the infrared small target image and the non-target image is set according to the ratio of 1:10 to 1:20 to construct an image sequence;

[0033] The subject wears an electroencephalogram cap, and the subject watches the image sequence in a rapid sequence visual presentation manner to obtain electroencephalogram data.

[0034] The electroencephalogram data is sample segmented to obtain samples; wherein the size of the sample is , represents the number of channels presented in the sample, and each channel represents an electrode, represents the sample duration.

[0035] The sample is filtered and standardized to obtain a processed sample as a training sample, and a training data set is constructed.

[0036] Specifically, in the embodiment, data is obtained through an RSVP experiment, that is, 8 subjects (5 males and 3 females, aged between 20-30 years old) wear an electroencephalogram cap and watch the image sequence in a rapid sequence visual presentation manner to obtain the electroencephalogram data of the subjects; wherein the subjects have normal or corrected-to-normal vision and no history of mental illness, the experiment follows the principles of the Helsinki Declaration, all subjects understand the experimental process and express their consent by signing the consent form. Infrared images from the LSOTB-TIR dataset are used in the experiment. First, images with infrared small targets are selected and adjusted to 1280x720 pixels to obtain 900 infrared small target images, please refer to Figure 3 , Figure 3 is a schematic diagram of an infrared small target image provided by the embodiment of the present application, and the target includes an airplane, an animal, a helicopter, a car and a person; please refer to Figure 4 , Figure 4 is a schematic diagram of a general RSVP paradigm provided by the embodiment of the present application, a text of the target type of the current test is displayed on the screen, a gaze is displayed for 1000 milliseconds to ensure that the attention of the subject is concentrated on the center of the screen, infrared small target images and non-target images are displayed in turn in the test, in each test, there are 45 images displayed at a rate of 5Hz, 2-4 infrared small target images exist in the 45 images, and the infrared small target images appear randomly; the whole experiment includes 18 blocks, and each block contains 12 tests. It should be noted that, within each block, in addition to the images prompted as targets, the remaining four types of images will serve as background images for the subject, and between every two tests, the subject is suggested to take a break. In the embodiment, the RSVP experimental paradigm code is realized by Psychopy.

[0037] The electroencephalogram data generated in the experimental process of the subject is recorded using a BioSemi-ActiveTwo system, which is based on the standard 10-20 system, the electroencephalogram cap has 64 scalp electrodes, the sampling rate is 1024Hz, and the impedance of each electrode is kept below 20kΩ to ensure high-quality signals. The collected electroencephalogram data is preprocessed in the following steps:

[0038] 1. Sample segmentation. For the collected EEG data, it is first segmented into EEG data samples, with each sample starting 1 second after the beginning of each stimulus. Each subject provides approximately 1280 EEG samples. For each sample, its size is where, denotes the number of electrodes presented in the sample, denotes the sample duration.

[0039] 2. Filtering and normalization. After sample segmentation, all samples are sent to a 6th order Butterworth band-pass filter with a cutoff frequency of 1-40 Hz, and then normalized using score method.

[0040] The selected public dataset contains RSVP EEG data from 8 healthy volunteers (7 females and 1 male, aged 19-29 years). The volunteers were asked to wear a non-invasive electroencephalogram acquisition device of BioSemi-ActiveTwo system, which is equipped with 256 scalp electrodes with a sampling frequency of 256 Hz. During data collection, the volunteers need to sit in front of a CRT screen in a dimly lit environment for 256 trials, with 50 image sequences presented at a frequency of 12 Hz, of which two are target images containing "airplanes". The EEG data of each trial is segmented into 1-second slices as training samples, resulting in a 256x256 matrix. These samples have undergone band-pass filtering with a cutoff frequency of 0.1-48 Hz and other data preprocessing.

[0041] In this experiment, the classification accuracy (acc) and standard deviation (std) are used as the standard for comparing the performance of different methods. Specifically, for the collected dataset, it contains EEG data from 8 healthy subjects, with each subject having approximately 1280 samples, including target and non-target samples. In order to divide the training dataset and the test dataset, 75% of the EEG data of each subject is selected as the training data, while the remaining 25% is designated as the test data. For the public dataset, the data of the remaining six subjects is used in subsequent experiments, as one subject only provides training data, and the data of another subject is damaged. In the public dataset, the division of the training dataset and the test dataset is predefined, and the final accuracy is obtained by calculating the average accuracy of each subject.

[0042] In this embodiment, the training dataset and the true labels of the training samples in the training dataset are input into the initial spatiotemporal progressive attention model, which is iteratively trained to extract the spatial and temporal features of the EEG data, including:

[0043] For the first The secondary iteration training process inputs a training data set into a progressive spatial learning module, extracts spatial features of electroencephalogram data, and obtains a spatial feature vector of the electroencephalogram data;

[0044] The spatial feature vector of the electroencephalogram data is input into a convolution layer and a first full connection layer for processing, and a low-dimensional spatial feature vector of the electroencephalogram data is obtained;

[0045] The low-dimensional spatial feature vector of the electroencephalogram data is input into a progressive temporal learning module, temporal features of the electroencephalogram data are extracted, and a temporal feature vector of the electroencephalogram data is obtained;

[0046] The temporal feature vector of the electroencephalogram data is input into a pooling layer for processing, and a spatial and temporal feature vector of the electroencephalogram data is obtained;

[0047] The spatial and temporal feature vector of the electroencephalogram data is input into a second full connection layer and a softmax function to map to a category space, and a predicted category is obtained.

[0048] In the embodiment, please refer to Figure 5 , Figure 5 is a schematic diagram of a progressive spatial learning module provided by the embodiment of the application, a training data set is input into the progressive spatial learning module, spatial features of electroencephalogram data are extracted, and a spatial feature vector of the electroencephalogram data is obtained, including:

[0049] The training sample is cut into multiple slices according to a time sequence, and the first slice is expressed as:

[0050] ;

[0051] Among them, represents data of the i-th channel (electrode) of the j-th slice; According to the j-th slice of the training sample, a first spatial undirected graph is constructed, and the expression is:

[0052]

[0053] ;

[0054] Among them, represents a set of nodes of the spatial undirected graph, represents a set of edges of the spatial undirected graph, represents an adjacency matrix representing the spatial relationship between the nodes of the spatial undirected graph according to the electrode position;

[0055] The first spatial undirected graph is input into a first graph convolution network of a first spatial expert in the progressive learning channel for processing, and a first spatial feature vector is obtained, and the expression is:​​​​

[0056] ;

[0057] wherein, represents an activation function, represents parameters to be learned in the graph convolution network, represents a normalized Laplacian matrix, represents a Chebyshev polynomial of order k, represents the dimension of the first spatial feature vector, represents the feature value in the first row and the first column of the feature obtained by the i-th expert;

[0058] According to the first spatial feature vector , a probability distribution predicted by the spatial expert is generated through a third fully connected layer, and the probability distribution predicted by the spatial expert is compared with the true label of the training sample to obtain a first loss function;

[0059] The partial linearization slope of the feature in the first column of the first spatial feature vector and the category is calculated, and according to the partial linearization slope, the importance weight of the feature in the first column is calculated through a first global average pooling layer, and the expression is as follows:

[0060] ;

[0061] wherein, represents the total number of channels, represents the feature value in the first row and the first column of the feature obtained by the i-th expert, represents a channel (i.e., an electrode), represents a partial derivative, represents a probability distribution predicted by the first expert for the i-th slice, represents the probability of the i-th class; The importance weight of all column features in the first spatial feature vector is multiplied by the corresponding column feature and summed to obtain the heat value of the electrode corresponding to the feature in the first row; through a first ReLU function, the electrode with a negative impact is excluded, and the expression is as follows:

[0062] ​​​​​​​​​​​

[0063] ;

[0064] Obtain the first spatial feature vector The calorific values ​​of the electrodes corresponding to all row features in the training samples are used to construct the first training sample. First spatial attention map of each slice ; first spatial attention map The value in and the first threshold For comparison, its expression is:

[0065] ;

[0066] Will be greater than the first threshold The electrodes corresponding to the values ​​are selected as important electrodes, and a first set of important electrodes is constructed. ;

[0067] The first spatial undirected graph is pruned using the first important electrode set to obtain the second spatial undirected graph. Second spatial experts then process the second spatial undirected graph to obtain the second spatial eigenvectors. At the same time, the second spatial attention map is obtained. Second important electrode set According to the second important electrode set Pruning the second-space undirected graph yields a third-space undirected graph. Third-space experts then process this third-space undirected graph to obtain third-space vectors. , the first spatial feature vector Second space feature vector and third space vector The merged spatial feature vector, output by the progressive learning module, is expressed as:

[0068] ;

[0069] The training samples Each slice is processed by the progressive learning module, and the output is... Each spatial feature vector, based on From the spatial feature vectors, we obtain the spatial feature vectors of the EEG data.

[0070] In this embodiment, the first loss function The expression is:

[0071] ;

[0072] ;

[0073] ;

[0074] in, Indicates the first Number of slices per training sample Indicates the first The true labels of each training sample Indicates that space experts have the right to the first The training sample of the th training sample The probability distribution of each slice prediction Indicates the first The probability of a class This represents the total number of training sample slices. This represents the total number of training samples.

[0075] In this embodiment, the spatial feature vector of EEG data is input into a convolutional layer and a first fully connected layer for processing to obtain a low-dimensional EEG data spatial feature vector, including:

[0076] Convolution and fully connected operations are performed on the spatial feature vectors of EEG data to reduce dimensionality and obtain low-dimensional spatial features. By sorting the data by time, we obtain the spatial feature vectors of low-dimensional EEG data. .

[0077] In this embodiment, please refer to Figure 6 , Figure 6 This is a schematic diagram of a progressive time learning module provided in an embodiment of the present invention. The module inputs a low-dimensional EEG data spatial feature vector to extract the temporal features of the EEG data, resulting in an EEG data temporal feature vector, including:

[0078] Based on the spatial feature vector of low-dimensional EEG data Constructing a first-time undirected graph Its expression is:

[0079] ;

[0080] in, It is an adjacency matrix that represents the temporal relationships between nodes in a temporal undirected graph based on the temporal location of the spatial characteristics of EEG data;

[0081] The first-time undirected graph is input into the second-graph convolutional network of the first-time expert in the progressive temporal learning module for processing, resulting in the first-time feature vector. Its expression is:

[0082] ;

[0083] in, This represents the activation function. This represents the parameters to be learned in a graph convolutional network. This represents the normalized Laplace matrix. express Chebyshev polynomials This represents the dimension of the first-time feature vector;

[0084] Based on the first time feature vector The probability distribution of time expert predictions is generated through the fourth fully connected layer. The probability distribution of time expert predictions is compared with the real labels of the training samples to obtain the second loss function.

[0085] The first-time feature vector is calculated through the second global average pooling layer. The Middle Weights of column features ; the first time feature vector The importance weights of all column features in the given data are multiplied by their corresponding column features and then summed to obtain the value of the first column feature. Heat value of time segment corresponding to row feature ;

[0086] Obtain the first time feature vector The heat values ​​of the electrodes corresponding to all row features are used to construct the first-time attention map of the training samples. The first time attention map The value in and the second threshold By comparing the values, the time segments corresponding to values ​​greater than the second threshold are identified as important segments, and a first set of important time segments is constructed.

[0087] The first-time undirected graph is pruned based on the first set of important time segments to obtain a second-time undirected graph. A second-time expert then processes the second-time undirected graph to obtain the second-time feature vector. At the same time, the second time attention map is obtained. The second important time segment set is used; the second time undirected graph is pruned based on the second important time segment to obtain the third time undirected graph; the third time expert processes the third time undirected graph to obtain the third time feature vector. The first time feature vector Second time feature vector and the third time feature vector The combined data is used as the time feature vector output by the progressive time learning module, i.e., the time feature vector of the EEG data.

[0088] In this embodiment, the expression for the second loss function is:

[0089] ;

[0090] ;

[0091] wherein, represents the true label of the i-th training sample.

[0092] In this embodiment, the electroencephalogram data time feature vector is input into the pooling layer for processing to obtain an electroencephalogram data space and time feature vector, and the expression thereof is:

[0093] ;

[0094] wherein, contains space and time information.

[0095] In this embodiment, the electroencephalogram data space and time feature vector is input into the second full connection layer and the softmax function to map to the category space to obtain a predicted category, including:

[0096] The electroencephalogram data space and time feature vector is flattened to obtain a flattened vector ;

[0097] The flattened vector is input into the second full connection layer and the softmax function for processing to obtain a logistic regression layer vector of probability distribution, and the expression thereof is:

[0098] ;

[0099] wherein, represents the number of categories, represents the weight, represents the bias;

[0100] The training sample is divided into the i-th category probability , and the expression thereof is:

[0101] ; wherein,

[0102] represents the value of the i-th category in ; The category corresponding to the highest probability is taken as the predicted category, and the expression thereof is:

[0103]

[0104] ;

[0105] wherein, represents the predicted category.

[0106] ​​​In summary, the application provides a fast sequence visual presentation electroencephalogram classification method based on a space-time progressive attention model, which has the following beneficial effects:

[0107] 1. The most important electrodes can be extracted; the progressive spatial learning (PSL) module of the application aims to extract electroencephalogram data features from more relevant electrodes, in order to achieve this goal, three spatial experts are constructed, the first spatial expert selects important electrodes and inputs the next spatial expert, and the process continues to filter out the electrodes most relevant to electroencephalogram data for the classification of electroencephalogram data for the RSVP task.

[0108] 2. Important time segments can be extracted; after obtaining the spatial features of all electroencephalogram data, the progressive temporal learning (PTL) module of the application aims to gradually select important electroencephalogram time segments to generate space-time electroencephalogram features for final classification. The application also uses the progressive attention to selectively focus on key time slices by training sample slices, which maximally reduces the influence of time difference of relevant potentials on classification.

[0109] 3. A specific RSVP paradigm is constructed; in order to solve the deficiency of the RSVP task application based on visible images, the application constructs a new data set IRED, which is composed of the electroencephalogram data of 8 subjects, and the electroencephalogram data is induced by weak infrared images with small targets; IRED is very helpful for exploring the application of the RSVP paradigm in low light or night environment, and enriches the diversity of stimuli for RSVP research.

[0110] It should be noted that in this article, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant are intended to cover non-exclusive inclusion, so that the article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the article or device including the element. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", and other directional or positional relationships indicated are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0111] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific feature or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0112] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the present application.

Claims

1. A method for RSVP electroencephalogram classification based on a spatiotemporal progressive attention model, characterized in that, The method comprises the following steps: obtaining brain electrical data to be classified; inputting the brain electrical data to be classified into a trained spatiotemporal progressive attention model for processing to obtain the category of the brain electrical data to be classified; wherein the trained spatiotemporal progressive attention model takes pre-set data collected and processed as a training data set, and is trained to extract spatial features and temporal features of the brain electrical data, and is obtained by training an initial spatiotemporal progressive attention model, wherein the pre-set data is obtained by collecting and processing images of a subject in a fast sequence visual presentation mode, and the images include infrared small target images and non-target images; before training the initial spatiotemporal progressive attention model, the method further comprises the following steps: constructing a spatiotemporal progressive attention model, which comprises a progressive spatial learning module, a convolution layer, a first full connection layer, a progressive temporal learning module, a pooling layer, a second full connection layer and a softmax function, the output end of the progressive spatial learning module is connected with the input end of the convolution layer, the output end of the convolution layer is connected with the input end of the first full connection layer, the output end of the first full connection layer is connected with the input end of the progressive temporal learning module, the output end of the progressive temporal learning module is connected with the input end of the pooling layer, the output end of the pooling layer is connected with the input end of the second full connection layer, and the softmax function is used to process the output result of the second full connection layer; wherein the progressive spatial learning module comprises a first spatial expert, a second spatial expert and a third spatial expert, the first spatial expert, the second spatial expert and the third spatial expert have the same structure and each comprises a first graph convolution network, a third full connection layer, a first global average pooling layer and a first ReLU function; the progressive temporal learning module comprises a first temporal expert, a second temporal expert and a third temporal expert, the first temporal expert, the second temporal expert and the third temporal expert have the same structure and each comprises a second graph convolution network, a fourth full connection layer, a second global average pooling layer and a second ReLU function.

2. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 1, characterized in that, The initial spatiotemporal progressive attention model is trained by taking the pre-set data collected and processed as a training data set and aiming to extract spatial features and temporal features of the brain electrical data, which comprises the following steps: obtaining brain electrical data to construct a training data set; wherein the training data set comprises a plurality of training samples; labeling the training samples in the training data set to obtain the true labels of the training samples; inputting the training data set and the true labels of the training samples in the training data set into the initial spatiotemporal progressive attention model for iterative training aiming to extract spatial features and temporal features of the brain electrical data, and obtaining the trained spatiotemporal progressive attention model when the loss function value reaches the minimum in the limited times of iterative training.

3. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 2, characterized in that, obtaining brain electrical data to construct a training data set, which comprises the following steps: obtaining infrared small target images and non-target images, setting the number of infrared small target images and non-target images in a ratio of 1:10 to 1:20 to construct an image sequence; The subject wears an electroencephalogram cap to watch the image sequence in a fast sequence visual presentation manner, and acquires electroencephalogram data of the subject; The electroencephalogram data is sample segmented to obtain samples; wherein, the size of the samples is , represents the number of channels presented in the sample, represents the sample duration; The sample is filtered and standardized to obtain a processed sample as a training sample, and a training data set is constructed.

4. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 2, characterized in that, The training data set and the true label of the training sample in the training data set are input into the initial spatiotemporal progressive attention model for iterative training for the purpose of extracting electroencephalogram data spatial features and temporal features, including: For the first iteration training process, the training data set is input to the progressive spatial learning module, the spatial features of the electroencephalogram data are extracted, and the electroencephalogram data spatial feature vector is obtained. The electroencephalogram data spatial feature vector is input into the convolution layer and the first full connection layer for processing to obtain a low-dimensional electroencephalogram data spatial feature vector; The low-dimensional electroencephalogram data spatial feature vector is input into the progressive time learning module to extract electroencephalogram data temporal features to obtain an electroencephalogram data temporal feature vector; The electroencephalogram data temporal feature vector is input into the pooling layer for processing to obtain an electroencephalogram data spatial and temporal feature vector; The electroencephalogram data spatial and temporal feature vector is input into the second full connection layer and the softmax function to map to the category space to obtain a predicted category.

5. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 4, characterized in that, The training data set is input into the progressive spatial learning module to extract electroencephalogram data spatial features to obtain an electroencephalogram data spatial feature vector, including: cutting the training samples into multiple slices according to a time sequence, a first slice is represented as: ​ ； wherein, represents the data of the i-th slice of the j-th channel. channel.​ According to the training sample's first Using slices, construct the first spatial undirected graph, whose expression is: ； wherein, represents a set of nodes of the spatial undirected graph, represents a set of edges of the spatial undirected graph, represents an adjacency matrix characterizing spatial relationships between nodes of the spatial undirected graph according to electrode positions; input the first space undirected graph to a first graph convolution network of a first space expert in the progressive space learning module for processing to obtain a first space feature vector The expression is: ； wherein, represents an activation function, represents parameters to be learned in the graph convolution network, represents a normalized Laplacian matrix, represents a Chebyshev polynomial of order k, represents the dimension of the spatial feature vector, represents the feature value of the i-th row and the j-th column in the feature obtained by the i-th spatial expert; represents the feature value of the i-th row and the j-th column in the feature obtained by the i-th spatial expert; represents the feature value of the i-th row and the j-th column in the feature obtained by the i-th spatial expert; represents the feature value of the i-th row and the j-th column in the feature obtained by the i-th spatial expert; According to the first spatial feature vector , a probability distribution of a spatial expert prediction is generated through a third fully connected layer, the probability distribution of the spatial expert prediction is compared with a true label of the training sample, and a first loss function is obtained. computing the first spatial feature vector in the middle column features and categories partial linearization slope, according to the partial linearization slope, the importance weight of the column feature is calculated by the first global average pooling layer column feature whose expression is: ； in, This indicates the total number of channels presented in the slice. Indicates the first The first feature obtained by the space expert Line number The characteristic values ​​of the column, Indicates a channel. This indicates finding the partial derivative. This indicates that the space experts have made a decision regarding the first... The probability distribution of each slice prediction Indicates the first The probability of a class; The first spatial feature vector The importance weight of each column feature in the matrix is multiplied by the corresponding column feature and summed up to obtain the first The heat value of the electrode corresponding to the row feature The electrodes with negative effects are excluded by the first ReLU function, and the expression is ； obtaining the first spatial feature vector all rows of the feature matrix correspond to the electrodes, and the first spatial attention map of the i-th slice of the training sample is constructed all rows of the feature matrix correspond to the electrodes, and the first spatial attention map of the i-th slice of the training sample is constructed ; comparing the value in the first spatial attention map with a first threshold value , and the expression is: ； corresponding to a value greater than the first threshold value as an important electrode, and constructing a first important electrode set corresponding to a value greater than the first threshold value as an important electrode, and constructing a first important electrode set ; The first spatial undirected graph is pruned using the first set of important electrodes to obtain a second spatial undirected graph. The second spatial expert then processes the second spatial undirected graph to obtain the second spatial feature vector. At the same time, the second spatial attention map is obtained. Second important electrode set According to the second important electrode set The second undirected graph is pruned to obtain a third undirected graph. The third-space expert then processes the third undirected graph to obtain a third-space vector. The first spatial feature vector The second spatial feature vector and the third space vector The features are merged and used as the spatial feature vector output by the progressive spatial learning module. The training samples are processed by the progressive spatial learning modules, and a plurality of spatial feature vectors are output. The spatial feature vectors are obtained according to the spatial feature vectors. The spatial feature vectors are obtained according to the spatial feature vectors. The spatial feature vectors are obtained according to the spatial feature vectors.

6. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 5, characterized in that, The first loss function The expression is: ； ； ； wherein, denotes the number of slices of the i-th training sample, denotes the true label of the i-th training sample, denotes the probability distribution of the j-th slice prediction of the i-th training sample by the spatial expert, denotes the probability of the i-th training sample belonging to the j-th class, denotes the probability of the i-th training sample belonging to the j-th class, denotes the total number of training sample slices, denotes the total number of training samples.​​​​ 7. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 4, characterized in that, The low-dimensional electroencephalogram data spatial feature vector is input into the progressive time learning module to extract electroencephalogram data temporal features to obtain an electroencephalogram data temporal feature vector, including: According to the low-dimensional brain electrical data space feature vector , a first time undirected graph is constructed , and its expression is: ； wherein, represents an adjacency matrix of the time-unoriented graph representing the temporal relationship between the nodes of the time-unoriented graph according to the temporal position of the spatial features of the electroencephalographic data; The first time undirected graph is input to a second graph convolution network of a first time expert in the progressive time learning module for processing to obtain a first time feature vector The expression is: ； wherein, denotes an activation function, denotes parameters to be learned in the graph convolution network, denotes a normalized Laplacian matrix, denotes Chebyshev polynomials of the first kind, denotes a dimension of the first time feature vector; According to the first time feature vector , a probability distribution of a time expert prediction is generated through a fourth fully connected layer, the probability distribution of the time expert prediction is compared with a true label of the training sample, and a second loss function is obtained. By the second global average pooling layer, the first time feature vector is calculated In the middle The weight of the column feature ; The importance weight of all column features in the first time feature vector is multiplied by the corresponding column feature and summed up to obtain the heat value of the time segment corresponding to the row feature ;​​ obtaining the first time feature vector the heat values of the electrodes corresponding to all the row features in the first time attention graph of the training sample comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a second threshold value comparing the values in the first time attention graph with a prune the first time undirected graph according to the first important time segment set to obtain a second time undirected graph, and the second time specialist processes the second time undirected graph to obtain a second time feature vector , and a second important time segment set; prune the second time undirected graph according to the second important time segment set to obtain a third time undirected graph, and the third time specialist processes the third time undirected graph to obtain a third time feature vector , and a second important time segment set; prune the second time undirected graph according to the second important time segment set to obtain a third time undirected graph, and the third time specialist processes the third time undirected graph to obtain a third time feature vector merge the first time feature vector , the second time feature vector , and the third time feature vector , as an output feature vector of the progressive time learning module.

8. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 7, characterized in that, The expression of the second loss function is: ； ； wherein, represents the true label of the i-th training sample. represents the true label of the i-th training sample.

9. The spatio-temporal progressive attention model-based RSVP electroencephalogram classification method according to claim 4, characterized in that, The electroencephalogram data spatial and temporal feature vector is input into the second full connection layer and the softmax function to map to the category space to obtain a predicted category, including: spatial and temporal feature vectors of the electroencephalographic data flattening, resulting in a flattened vector ; The flattened vector is input into the second full connection layer and the softmax function for processing to obtain a logistic regression layer vector of probability distribution, and the expression is: ； wherein, represents the number of classes, represents a weight value, represents a bias; The training sample divided into first Class probability The expression is: ； wherein represents the value of the class; The category corresponding to the highest probability is taken as the predicted category, and the expression is: ； wherein, represents a predicted class.

Images