A cognitive classification method, system and medium based on EEG spatiotemporal semantic features

Abstract

The present invention discloses a cognitive classification method, system and medium based on EEG spatiotemporal semantic features. The present invention includes using a pre-trained EEG signal classification network to realize cognitive task classification. The EEG signal classification network includes a semantic unit encoding module, a temporal semantic feature extraction module and a spatial semantic feature learning module connected in sequence. The semantic unit encoding module is used to extract the underlying features of the pre-processed EEG signal and encode the semantic unit to obtain the semantic unit. The temporal semantic feature extraction module is used to extract the temporal semantic features of the semantic unit to obtain the temporal semantic features. The spatial semantic feature learning module is used to learn the spatial topological relationship of the temporal semantic features to obtain the spatiotemporal semantic features. The present invention aims to utilize the characteristics of rich semantic features and strong spatiotemporal representation of EEG signals to realize the extraction of representative EEG spatiotemporal semantic features and improve the accuracy of cognitive task classification.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to a cognitive classification method, system and medium based on EEG spatiotemporal semantic features. Background Art

[0002] EEG signals can reflect various neural activities of the brain. Because of their objectivity, non-invasiveness, and ease of operation, they have become the preferred modality for different brain-computer interface task paradigms. EEG signals have the characteristics of high temporal resolution and can capture temporal changes on a very small time scale. However, their spatial resolution is relatively low, and different operators choose different electrode placement standards when collecting EEG signals, making it difficult to extract universal EEG spatial patterns. In addition, the EEG signal-to-noise ratio is low and is greatly affected by noise interference. Therefore, how to analyze and decode EEG signals to achieve effective feature extraction is a challenging problem. Most of the existing end-to-end EEG feature extraction methods based on deep learning directly extract features from raw EEG data. In fact, for different cognitive tasks, there is semantic information related to prior knowledge in EEG signals. For example, in the motor imagery task, different brain regions have different activation levels, and each brain region can be regarded as an independent unit with a specific activation level. Since these units contain physiological knowledge such as brain region division and semantic information, these brain regions can be called spatial semantic units. The spatial pattern composed of these spatial semantic units with different activation levels contains spatial features related to the task. Similarly, short-time slices of EEG data can also be called temporal semantic units. The temporal dependencies between units contain task-related temporal features. How to extract the underlying features that can characterize semantic units from the original signal and further extract the spatiotemporal dependencies of semantic units to obtain spatiotemporal semantic features has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a cognitive classification method, system and medium based on EEG spatiotemporal semantic features are provided. The present invention aims to utilize the characteristics of EEG signals such as rich semantic features and strong spatiotemporal representation to realize the extraction of representative EEG spatiotemporal semantic features and improve the accuracy of cognitive task classification.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A cognitive classification method based on EEG spatiotemporal semantic features, comprising using a pre-trained EEG signal classification network to implement cognitive task classification on pre-processed EEG signals, wherein the EEG signal classification network comprises an interconnected EEG spatiotemporal semantic feature extraction network and a classification network, wherein the EEG spatiotemporal semantic feature extraction network comprises a sequentially connected semantic unit encoding module, a temporal semantic feature extraction module and a spatial semantic feature learning module, wherein the semantic unit encoding module is used to extract underlying features from pre-processed EEG signals and perform semantic unit encoding to obtain semantic units, wherein the temporal semantic feature extraction module is used to perform temporal semantic feature extraction on semantic units to obtain temporal semantic features, wherein the spatial semantic feature learning module is used to perform spatial topological relationship learning on temporal semantic features to obtain spatiotemporal semantic features, and wherein the classification network is used to classify spatiotemporal semantic features to obtain cognitive classification results of EEG signals.

[0006] Optionally, the semantic unit encoding module extracts underlying features from the preprocessed EEG signal and performs semantic unit encoding to obtain semantic units including:

[0007] S101, extracting multiple features from the preprocessed EEG signal using multi-scale convolution;

[0008] S102, connecting multiple features by splicing along the feature channel dimension to obtain a dynamic time feature;

[0009] S103, performing feature fusion on the dynamic time features to obtain fusion features;

[0010] S104, introducing a weight matrix of the same size as the fusion feature, performing a dot product between the weight matrix and the fusion feature to implement the allocation of the attention mechanism to the important elements in the fusion feature, wherein the weight matrix is ​​a trainable parameter matrix;

[0011] S105, performing a dot product operation to obtain features of each electrode channel, and calculating the feature averages of each electrode channel in each brain region in units of brain regions to implement a brain region-based aggregation operation;

[0012] S106, the underlying features output by the aggregation operation are sliced ​​according to time, so that each unit element represents a different divided brain area in the spatial dimension, and each unit element represents a time slice in the time dimension, thereby dividing the underlying features into semantic units.

[0013] Optionally, the multi-scale convolution in step S101 includes multiple feature extraction network branches, each feature extraction network branch includes a convolution layer and an average pooling layer connected in sequence, and the convolution layer in each feature extraction network branch has convolution kernels of different scales.

[0014] Optionally, in step S103, fusing the dynamic time features means fusing the dynamic time features sequentially through a depthwise separable convolutional layer and an average pooling layer, and the output of the depthwise separable convolutional layer serves as the input of the average pooling layer.

[0015] Optionally, when the characteristic average values ​​of each electrode channel of each brain region are calculated in units of brain regions in step S105 to realize the aggregation operation based on brain regions, the brain region division mode is {Fp1, Fpz, Fp2}, {AF7, AFF5h, AF3,AFz, AF4, AFF6h, AF8}, {F7, F5, F3, F1, Fz, F2, F4, F6, F8}, {FC5, FC3, FC1,FCz, FC2, FC4, FC6}, {C5, C3, C1, Cz, C2, C4, C6}, {CP5, CP3, CP1, CPz, CP2,CP4, CP6}, {P7, P5, P3, P1, Pz, P2, P4, P6, P8}, {PO7, PO3, POz, PO4, PO8}, {O1,Oz, O2}, {FT7, T7, TP7}, {FT8, T8, TP8}, where each set of curly brackets represents a divided brain area, and the elements in the curly brackets are the names of the electrode channels.

[0016] Optionally, the temporal semantic feature extraction module performs temporal semantic feature extraction on the semantic unit to obtain the temporal semantic feature, including: using a time domain convolutional network TCN to perform temporal semantic feature extraction on the semantic unit of each brain region to obtain the temporal semantic feature, the time domain convolutional network TCN includes multiple levels of feature extraction modules connected in sequence, each level of feature extraction module includes a multi-level extended causal convolution module, a ReLU activation function layer and a random dropout layer, and the input features of the feature extraction module and the output features of the multi-level extended causal convolution module, the ReLU activation function layer and the random dropout layer are added through jump connections and then passed through a ReLU activation function layer to obtain the output features of the feature extraction module.

[0017] Optionally, the spatial semantic feature learning module performs spatial topological relationship learning on the temporal semantic feature to obtain the spatiotemporal semantic feature, including:

[0018] S201, calculating the self-similarity of the temporal semantic feature according to the following formula:

[0019] ,

[0020] in, is the self-similarity of temporal semantic features, for Activation function, For the temporal semantic features; construct a feature self-similarity matrix according to the self-similarity of each temporal semantic feature;

[0021] S202, adding the feature self-similarity matrix, the parameter learnable matrix and the identity matrix to obtain an adaptive matrix, wherein the parameter learnable matrix is ​​a trainable parameter matrix;

[0022] S203, a graph structure is constructed by taking brain regions as graph nodes, temporal semantic features of brain regions as node attributes, and adaptive matrices as adjacency matrices, and graph convolution GCN is used to extract spatial semantic features from the graph structure.

[0023] In addition, the present invention also provides a cognitive classification system based on EEG spatiotemporal semantic features, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features.

[0024] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features through a processor.

[0025] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features through a processor.

[0026] Compared with the prior art, the present invention mainly has the following advantages: 1. The present invention includes using a pre-trained EEG signal classification network to realize cognitive task classification of pre-processed EEG signals. The EEG signal classification network includes a semantic unit encoding module, a temporal semantic feature extraction module and a spatial semantic feature learning module connected in sequence. The semantic unit encoding module is used to extract the underlying features of the pre-processed EEG signals and encode the semantic units to obtain semantic units. The temporal semantic feature extraction module is used to extract temporal semantic features of the semantic units to obtain temporal semantic features. The spatial semantic feature learning module is used to learn the spatial topological relationship of the temporal semantic features to obtain spatiotemporal semantic features. The present invention aims to utilize the characteristics of EEG signals such as rich semantic features and strong spatiotemporal representation to realize the extraction of representative EEG spatiotemporal semantic features and improve the accuracy of cognitive task classification. 2. The present invention designs a semantic unit encoding module to extract the underlying features of the pre-processed EEG signals and encode the semantic units to obtain semantic units to represent the learned task-related physiological knowledge. 3. The present invention proposes a semantic feature architecture, which uses a spatial semantic feature learning module to learn the spatial topological relationship of temporal semantic features to obtain spatiotemporal semantic features, and can extract the spatiotemporal semantic relationship of semantic units under different cognitive tasks. 4. The method of the present invention is suitable for cognitive state assessment experiments of different cognitive tasks and has strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a basic flow chart of the method of the embodiment of the invention.

[0028] Figure 2 Schematic diagram of the network structure of the EEG signal classification network in the embodiment of the invention.

[0029] Figure 3 Schematic diagram of the network structure of the semantic unit encoding module in the embodiment of the invention.

[0030] Figure 4 Schematic diagram of the network structure of the temporal semantic feature extraction module and the time domain convolutional network TCN in an embodiment of the invention, wherein (a) is a schematic diagram of the network structure of the temporal semantic feature extraction module, and (b) is a schematic diagram of the network structure of a single time domain convolutional network TCN.

[0031] Figure 5 Schematic diagram of the network structure of the feature extraction module in the time domain convolutional network TCN in an embodiment of the invention.

[0032] Figure 6 Schematic diagram of the network structure of the spatial semantic feature learning module in the embodiment of the invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the cognitive classification method based on EEG spatiotemporal semantic features of this embodiment includes using a pre-trained EEG signal classification network to implement cognitive task classification using preprocessed EEG signals, including: building and training a deep learning model based on deep learning (i.e., an EEG signal classification network), the EEG signal classification network includes an interconnected EEG spatiotemporal semantic feature extraction network and a classification network; collecting EEG signals of subjects completing cognitive tasks; preprocessing the EEG signals; and inputting the EEG signals into the deep learning model to obtain evaluation results of cognitive tasks.

[0035] When the subjects perform different cognitive tasks, different cognitive states will be induced, and the cerebral cortex will produce neural electrical signals associated with cognitive tasks and cognitive states. The method of extracting EEG spatiotemporal semantic features in this embodiment selects EEG signals of subjects when completing cognitive tasks for feature extraction to achieve cognitive state assessment, without specifying specific cognitive tasks, in order to verify the generalization ability of the EEG feature extraction method proposed in the present invention. Therefore, the cognitive tasks of the cognitive classification method based on EEG spatiotemporal semantic features in this embodiment are not limited to a specific cognitive task.

[0036] In this embodiment, preprocessing the EEG signal includes: downsampling the EEG signal; performing baseline correction on the downsampled EEG signal by removing the channel mean; using a bandpass filter to remove noise interference from the baseline-corrected EEG signal; and windowing the filtered EEG signal.

[0037] like Figure 2As shown, the EEG signal classification network in this embodiment includes an interconnected EEG spatiotemporal semantic feature extraction network and a classification network, wherein the EEG spatiotemporal semantic feature extraction network includes a semantic unit encoding module, a temporal semantic feature extraction module and a spatial semantic feature learning module which are sequentially connected, wherein the semantic unit encoding module is used to extract underlying features of the preprocessed EEG signal (c represents the number of EEG channels, and T represents the number of EEG time samples) and perform semantic unit encoding to obtain a semantic unit, wherein the temporal semantic feature extraction module is used to perform temporal semantic feature extraction on the semantic unit to obtain temporal semantic features, wherein the spatial semantic feature learning module is used to perform spatial topological relationship learning on the temporal semantic features to obtain spatiotemporal semantic features, and the classification network is used to classify the spatiotemporal semantic features to obtain cognitive classification results of the EEG signal.

[0038] In this embodiment, the preprocessed EEG signal is input into the semantic unit encoding module to extract its underlying features, and the semantic unit encoding includes: using multi-scale convolution to extract the dynamic time features of the EEG signal, and then fusing the dynamic time features, based on the physiological knowledge of brain functional areas and the attention mechanism, channel division and attention allocation are performed on the fused features to obtain the semantic unit; specifically, Figure 3 As shown, the semantic unit encoding module in this embodiment extracts the underlying features of the preprocessed EEG signal and performs semantic unit encoding to obtain the semantic units including:

[0039] S101, extracting multiple features from the preprocessed EEG signal using multi-scale convolution;

[0040] S102, connecting multiple features by splicing along the feature channel dimension to obtain a dynamic time feature;

[0041] S103, performing feature fusion on the dynamic time features to obtain fusion features;

[0042] S104, introducing a weight matrix of the same size as the fusion feature, performing a dot product between the weight matrix and the fusion feature to implement the allocation of the attention mechanism to the important elements in the fusion feature, wherein the weight matrix is ​​a trainable parameter matrix;

[0043] S105, performing a dot product operation to obtain features of each electrode channel, and calculating the feature averages of each electrode channel in each brain region in units of brain regions to implement a brain region-based aggregation operation;

[0044] S106, the underlying features output by the aggregation operation are sliced ​​according to time, so that each unit element represents a different divided brain area in the spatial dimension, and each unit element represents a time slice in the time dimension, thereby dividing the underlying features into semantic units. Figure 3In the above figure, c represents the number of EEG channels, t represents the number of time samples of fused features, s represents the number of feature channels (the number of convolution kernels), and J represents the number of brain regions.

[0045] The multi-scale convolution in step S101 of this embodiment includes multiple feature extraction network branches, each of which includes sequentially connected convolution layers and average pooling layers, and the convolution layers in each feature extraction network branch have convolution kernels of different scales. In this embodiment, the multi-scale convolution specifically includes three feature extraction network branches, and the EEG signal is feature extracted using three feature extraction network branches. The feature branches include connected convolution layers and average pooling layers, and the convolution layers have convolution kernels of different scales in different feature extraction network branches. The output of the convolution layer serves as the input of the average pooling layer; the features extracted by the three branches are connected in a splicing manner to obtain dynamic time features.

[0046] In this embodiment, the feature fusion of the dynamic time features in step S103 refers to fusing the dynamic time features in sequence through a depthwise separable convolutional layer and an average pooling layer, and the output of the depthwise separable convolutional layer is used as the input of the average pooling layer.

[0047] In this embodiment, when the characteristic average values ​​of each electrode channel of each brain region are calculated in step S105 to realize the aggregation operation based on brain regions, the brain functional regions are divided according to the physiological knowledge of brain functional regions, and the brain region division mode is {Fp1, Fpz, Fp2}, {AF7, AFF5h, AF3, AFz, AF4, AFF6h, AF8}, {F7, F5,F3, F1, Fz, F2, F4, F6, F8}, {FC5, FC3, FC1, FCz, FC2, FC4, FC6}, {C5, C3, C1,Cz, C2, C4, C6}, {CP5, CP3, CP1, CPz, CP2, CP4, CP6}, {P7, P5, P3, P1, Pz, P2,P4, P6, P8}, {PO7, PO3, POz, PO4, PO8}, {O1, Oz, O2}, {FT7, T7, TP7}, {FT8, T8,TP8}, where each set of curly brackets represents a divided brain area, and the elements in the curly brackets (such as Fp1, Fpz) are the names of electrode channels.

[0048] like Figure 4 As shown in (a) in FIG. 1 , the temporal semantic feature extraction module in this embodiment is composed of multiple temporal convolutional networks (TCN). Figure 4In , t represents the number of time samples of fusion features, and J represents the number of brain regions (equal to the number of time domain convolutional networks TCN). Figure 4 As shown in (b), each time domain convolution network TCN extracts time semantic features from the semantic units representing the same brain region. In this embodiment, the time semantic feature extraction module extracts time semantic features from the semantic units to obtain time semantic features, including: extracting time semantic features from the semantic units of each brain region using a time domain convolution network TCN to obtain time semantic features, wherein the time domain convolution network TCN includes multiple levels of feature extraction modules connected in sequence.

[0049] like Figure 5 As shown, in this embodiment, each level of feature extraction module includes a multi-level extended causal convolution module, a ReLU activation function layer and a random drop layer, and the input features of the feature extraction module and the output features of the multi-level extended causal convolution module, the ReLU activation function layer and the random drop layer are added through a jump connection and then passed through a ReLU activation function layer to obtain the output features of the feature extraction module.

[0050] like Figure 6 As shown, in this embodiment, the spatial semantic feature learning module learns the spatial topological relationship of the temporal semantic feature to obtain the spatiotemporal semantic feature, including:

[0051] S201, calculating the self-similarity of the temporal semantic feature according to the following formula:

[0052] ,

[0053] in, is the self-similarity of temporal semantic features, for Activation function, For the temporal semantic features; construct a feature self-similarity matrix according to the self-similarity of each temporal semantic feature;

[0054] S202, adding the feature self-similarity matrix, the parameter learnable matrix and the identity matrix to obtain an adaptive matrix, wherein the parameter learnable matrix is ​​a trainable parameter matrix;

[0055] S203, a graph structure is constructed by taking brain regions as graph nodes, temporal semantic features of brain regions as node attributes, and adaptive matrix as adjacency matrix, and graph convolutional networks (GCN) is used to extract spatial semantic features from the graph structure.

[0056] The classification network for spatiotemporal semantic features can adopt the required network model as needed. For example, as an optional implementation, the classification network in this embodiment is composed of a fully connected layer and a softmax classifier connected in series.

[0057] In order to verify the cognitive classification method based on the spatiotemporal semantic features of EEG in this embodiment, four public data sets (Attention, DEAP-V, DEAP-L, Fatigue) are used in this embodiment to evaluate and compare cognitive states. Attention data set: The EEG signals of 26 subjects when performing discrimination and selection tasks are collected, with a sampling frequency of 1000Hz. The task flow is as follows: The first two seconds are the instructions of the task. The next 40 seconds are the task period, and a symbol "O" and "X" appear randomly every 2 seconds. The subject judges and presses the corresponding button. In the last 20 seconds, the subject is required to stare at the fixed cross in the middle of the display for testing. The correct experimental trial is selected for binary classification attention evaluation. DEAP-V data set: The EEG signals of 32 subjects when watching 40 one-minute music video clips are collected, with a sampling frequency of 512Hz. The subjects completed the questionnaire, scored each video clip independently, selected the Valence value as the classification indicator of the emotional task, and made a binary classification label based on the scoring results to achieve the classification of different emotional states of the subjects. DEAP-L dataset: The DEAP-L dataset uses the same experimental paradigm as the DEAP-V dataset to collect EEG signals and obtain the same EEG data, but selects the liking value in the subject questionnaire as the label to achieve binary classification of the preference task. Fatigue dataset: Collects EEG data from 27 subjects while performing a four-lane simulated highway driving task for about 90 minutes. The subjects were asked to drive the car back to the original cruising lane when the car deviated to another lane. The subject's reaction time was used to make a binary classification label. The sampling frequency is 500Hz.

[0058] In order to verify the performance of cognitive classification method based on EEG spatiotemporal semantic features in cognitive state assessment on different cognitive tasks, it is compared with the existing advanced models, including DeepConvNet, EEGNet, TSception, ACRNN, AMCNN-DGCN, EEG-ViT, R2G-STNN, LCA, HRNN, HSLT, RGNN, GraphNet, EEG-Conformer, LGGNet, EEGAT and EEG-Deformer. Finally, the results of the method of this embodiment and other existing advanced models are shown in Table 1.

[0059] Table 1 Comparison of the results of this embodiment method with other existing advanced models

[0060]

[0061] As shown in Table 1, the method of this embodiment is superior to other existing methods in terms of accuracy and F1 value indicators in four cognitive tasks. The comparison results prove the effectiveness of the method of this embodiment.

[0062] In order to evaluate the contribution of each module in the EEG signal classification network of the method of this embodiment, this embodiment designs a series of ablation experiments, and the evaluation results are shown in Table 2.

[0063] Table 2 Ablation study based on attention task evaluation

[0064]

[0065] See Table 2, the model evaluation results proposed by the method of this embodiment are shown in the last row of the table. The ablation experiment analysis in this embodiment is as follows: 1) The impact of the semantic unit encoding module on the evaluation results: First, the multi-scale convolution layer is replaced with the same-scale convolution layer; from the first row of Table 2, the accuracy and F1 score drop from 87.55% and 87.54% to 87.10% and 87.21%, respectively, which fully proves that the multi-scale convolution layer can learn more diverse temporal information. From the second row of Table 2, it can be seen that when the multi-scale convolution layer is retained and the feature fusion layer is removed, the accuracy and F1 score drop by 5.30% and 6.03%, respectively, which shows that the spliced ​​multi-scale temporal features are not fully coupled and it is difficult to further extract relevant features. The attention mechanism is removed, that is, no prior information is introduced to encode the underlying features, and the results are shown in the third row of Table 2. The performance of the ablation model is far inferior to the method of this embodiment, with a 7.24% drop in accuracy and a 7.29% drop in comprehensive evaluation indicators. The sharp drop in the index proves that prior knowledge is important for encoding semantic units, which can distinguish more important temporal and spatial units for subsequent feature extraction. 2) The impact of the temporal semantic feature extraction module on the evaluation results: All TCN networks in the module are removed to analyze the ability of the method to extract temporal semantic features. Each semantic unit representing the same brain area is directly used as the attribute of the node in the global topology map. As shown in the fourth row of Table 2, both evaluation indicators dropped by 1.25%, indicating that it is not enough to extract only the underlying temporal features of the original EEG data. The semantic temporal feature extraction module can capture temporal features from a more macro perspective, thereby improving the evaluation performance of the model. 3) The impact of the spatial semantic feature learning module on the evaluation results: By removing the parameter learnable matrix and the feature self-similarity matrix respectively, it can be found that the accuracy index dropped by 0.30% and 1.07%. When retaining the prior adjacency relationship, the performance of the method is better than retaining the learnable adjacency relationship, proving that the prior adjacency relationship contains more potential spatial semantic information. It can be seen that the cognitive classification method based on EEG spatiotemporal semantic features in this embodiment aims to solve the problem of insufficient extraction of spatiotemporal relationship of semantic features of EEG signals, and builds and trains a neural network model based on semantic unit encoding module, temporal semantic feature extraction module and spatial semantic feature learning module; the obtained spatiotemporal semantic feature extraction method has achieved better performance in different cognitive task evaluations, and has the advantages of high evaluation accuracy and strong generalization performance.

[0066] In addition, the present invention also provides a cognitive classification system based on EEG spatiotemporal semantic features, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features.

[0067] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features through a processor.

[0068] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features through a processor.

[0069] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A cognitive classification method based on EEG spatiotemporal semantic features, characterized in that: The method comprises using a pre-trained EEG signal classification network to realize cognitive task classification of the pre-processed EEG signal, wherein the EEG signal classification network comprises an EEG spatiotemporal semantic feature extraction network and a classification network connected to each other, wherein the EEG spatiotemporal semantic feature extraction network comprises a semantic unit encoding module, a temporal semantic feature extraction module and a spatial semantic feature learning module connected in sequence, wherein the semantic unit encoding module is used to extract the underlying features of the pre-processed EEG signal and perform semantic unit encoding to obtain a semantic unit, wherein the temporal semantic feature extraction module is used to perform temporal semantic feature extraction on the semantic unit to obtain a temporal semantic feature, wherein the spatial semantic feature learning module is used to perform spatial topological relationship learning on the temporal semantic feature to obtain a spatiotemporal semantic feature, and wherein the classification network is used to classify the spatiotemporal semantic features to obtain a cognitive classification result of the EEG signal; The semantic unit encoding module extracts the underlying features of the preprocessed EEG signal and encodes the semantic units to obtain the following semantic units: S101, extracting multiple features from the preprocessed EEG signal using multi-scale convolution; S102, connecting multiple features by splicing along the feature channel dimension to obtain a dynamic time feature; S103, performing feature fusion on the dynamic time features to obtain fusion features; S104, introducing a weight matrix of the same size as the fusion feature, performing a dot product between the weight matrix and the fusion feature to implement the allocation of the attention mechanism to the important elements in the fusion feature, wherein the weight matrix is ​​a trainable parameter matrix; S105, performing a dot product operation to obtain features of each electrode channel, and calculating the feature averages of each electrode channel in each brain region in units of brain regions to implement a brain region-based aggregation operation; S106, slicing the underlying features output by the aggregation operation according to time, so that each unit element represents a different divided brain area in the spatial dimension, and each unit element represents a time slice in the time dimension, thereby dividing the underlying features into semantic units; The spatial semantic feature learning module learns the spatial topological relationship of the temporal semantic features to obtain spatiotemporal semantic features including: S201, calculating the self-similarity of the temporal semantic feature according to the following formula: ， in, is the self-similarity of temporal semantic features, for Activation function, For the temporal semantic features; construct a feature self-similarity matrix according to the self-similarity of each temporal semantic feature; S202, adding the feature self-similarity matrix, the parameter learnable matrix and the identity matrix to obtain an adaptive matrix, wherein the parameter learnable matrix is ​​a trainable parameter matrix; S203, a graph structure is constructed by taking brain regions as graph nodes, temporal semantic features of brain regions as node attributes, and adaptive matrices as adjacency matrices, and graph convolution GCN is used to extract spatial semantic features from the graph structure.

2. The cognitive classification method based on EEG spatiotemporal semantic features according to claim 1 is characterized in that: The multi-scale convolution in step S101 includes multiple feature extraction network branches, each of which includes a convolution layer and an average pooling layer connected in sequence, and the convolution layer in each feature extraction network branch has convolution kernels of different scales.

3. The cognitive classification method based on EEG spatiotemporal semantic features according to claim 1 is characterized in that: In step S103, the feature fusion of the dynamic time features refers to fusing the dynamic time features through a depthwise separable convolution layer and an average pooling layer in sequence, and the output of the depthwise separable convolution layer is used as the input of the average pooling layer.

4. The cognitive classification method based on EEG spatiotemporal semantic features according to claim 1, characterized in that: In step S105, when the characteristic average values ​​of each electrode channel of each brain region are calculated in units of brain regions to realize the aggregation operation based on brain regions, the brain region division mode is {Fp1, Fpz, Fp2}, {AF7, AFF5h, AF3, AFz, AF4, AFF6h,AF8}, {F7, F5, F3, F1, Fz, F2, F4, F6, F8}, {FC5, FC3, FC1, FCz, FC2, FC4,FC6}, {C5, C3, C1, Cz, C2, C4, C6}, {CP5, CP3, CP1, CPz, CP2, CP4, CP6}, {P7,P5, P3, P1, Pz, P2, P4, P6, P8}, {PO7, PO3, POz, PO4, PO8}, {O1, Oz, O2}, {FT7,T7, TP7}, {FT8, T8, TP8}, where each set of curly brackets represents a divided brain area, and the elements in the curly brackets are the names of the electrode channels.

5. The cognitive classification method based on EEG spatiotemporal semantic features according to claim 1, characterized in that: The temporal semantic feature extraction module extracts temporal semantic features from semantic units to obtain temporal semantic features, including: extracting temporal semantic features from the semantic units of each brain region using a time domain convolutional network TCN to obtain temporal semantic features, wherein the time domain convolutional network TCN includes multiple levels of feature extraction modules connected in sequence, each level of feature extraction module includes a multi-level extended causal convolution module, a ReLU activation function layer, and a random dropout layer, and the input features of the feature extraction module and the output features of the multi-level extended causal convolution module, the ReLU activation function layer, and the random dropout layer are added through jump connections and then passed through a ReLU activation function layer to obtain the output features of the feature extraction module.

6. A cognitive classification system based on EEG spatiotemporal semantic features, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features as described in any one of claims 1 to 5 through a processor.

8. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the cognitive classification method based on EEG spatiotemporal semantic features as described in any one of claims 1 to 5 through a processor.

Images