EEG decoding method and system based on global attention and recurrent neural network

Abstract

The present invention discloses an EEG decoding method and system based on global attention and recurrent neural networks. The method comprises dividing the original EEG signal segment to be decoded into sample sub-segments; extracting shallow features from the sample sub-segments; fusing and dimensionalizing the shallow features; sequentially performing spatial feature extraction, dimensional compression, and temporal feature extraction on the fused and dimensionally unified shallow features based on global attention and recurrent neural networks to obtain EEG spatiotemporal features; implementing feature extraction using a globally reinforced recurrent neural network model composed of multiple sets of cascaded position encoding modules, encoder layers, and recurrent neural network models; and classifying the EEG spatiotemporal features to obtain EEG decoding results. The present invention aims to overcome the drawback of recurrent neural networks, which have difficulty in fully extracting global spatiotemporal information, and to fully enhance the network's ability to extract spatiotemporal features of EEG signals and improve decoding accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram (EEG) signal processing, and in particular relates to an EEG decoding method and system based on global attention and recurrent neural networks. Background Art

[0002] In recent years, deep learning-based EEG decoding has made significant progress. However, effectively extracting spatiotemporal information remains a significant challenge. Current methods for extracting spatial information primarily rely on convolution operations. However, relationships between electrodes are not limited to adjacent electrodes; significant correlations can also exist between distant electrodes, making convolutional methods difficult to effectively capture such dependencies. To address this, some researchers have introduced recurrent neural networks, particularly long-short-term memory (LSTM) networks, leveraging their ability to capture long-term and short-term dependencies to extract interelectrode relationships, achieving some progress. However, a deeper analysis of the computational mechanism of RNNs reveals that information is input to units in a sequential manner, and each unit can only access information sequentially. This makes it difficult for electrodes arranged within each RNN unit to fully access the global information of other units, effectively destroying interelectrode correlations. Regarding temporal feature extraction, EEG samples are typically divided into multiple time segments, and LSTM networks are used to capture the correlations between different time points. Similarly, such methods struggle to extract global temporal correlations between different time points in an EEG sample. Therefore, how to overcome the defect that recurrent neural networks have difficulty in fully extracting global spatiotemporal information, and further improve the spatiotemporal information extraction capability of EEG signals and improve decoding accuracy, is a key technical problem that needs to be solved urgently. Summary of the Invention

[0003] Technical problem to be solved by the present invention: In response to the above-mentioned problems of the prior art, a method and system for EEG decoding based on global attention and recurrent neural network are provided. The present invention aims to overcome the defect that recurrent neural network has difficulty in fully extracting global spatiotemporal information, fully improve the network's ability to extract spatiotemporal features of EEG signals, and improve decoding accuracy.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A method for decoding EEG based on global attention and recurrent neural network, comprising the following steps:

[0006] Divide the original EEG signal segment to be decoded into sample sub-segments;

[0007] Extract shallow features from sample sub-segments;

[0008] Fusion of shallow features and unification of dimensions;

[0009] Based on global attention and recurrent neural network, the shallow features after fusion and dimensionality unification are sequentially subjected to spatial feature extraction, dimensionality compression and temporal feature extraction to obtain EEG spatiotemporal features; wherein both spatial feature extraction and temporal feature extraction are implemented by adopting a globally enhanced recurrent neural network model, the globally enhanced recurrent neural network model includes a multi-stage cascade unit, the cascade unit is composed of a position encoding module, an encoder layer and a recurrent neural network model connected in sequence, the encoder layer is an encoder using a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement, and when performing spatial feature extraction, the position encoding module embeds the position encoding of the electrode channel of the input data in a preset two-dimensional table of electrode channel positions, and when performing temporal feature extraction, the position encoding module embeds the sequential encoding of the sample sub-segment in the original sample segment;

[0010] Classify the EEG spatiotemporal features to obtain the EEG decoding results.

[0011] Optionally, dividing the original EEG signal segments to be decoded into samples includes: dividing the original EEG signal segments into samples according to the time length Divide into several sample segments, and then divide the sample segments into smaller preset durations The shape obtained by division is The sample sub-segment of is the number of electrode channels in the original EEG signal, is the downsampling rate of the EEG signal.

[0012] Optionally, extracting shallow features from the sample sub-segment includes: dividing the data of each electrode channel in the sample sub-segment into different frequency band windows, using a bandpass filter to filter based on a frequency window of a specified size to obtain a plurality of different frequency band signals, and calculating the differential entropy feature of each frequency band signal to obtain a dimension of The differential entropy features are used as the extracted shallow features.

[0013] Optionally, the fusion of shallow features and dimension unification includes: inputting shallow features with dimensions of as well as The fusion and dimension unification module composed of two fully connected layers FC is used to unify the shallow feature shapes into ,in is the number of electrode channels in the original EEG signal, is a dimension parameter, and the activation function used between the two fully connected layers FC is the linear rectification function Relu.

[0014] Optionally, when the shallow features after fusion and dimensionality unification based on global attention and recurrent neural network are sequentially subjected to spatial feature extraction, dimensionality compression and time feature extraction to obtain EEG spatiotemporal features, the spatial feature extraction and dimensionality compression include: for the shape of The shallow features of is the number of electrode channels in the original EEG signal, As the dimension parameter, the global enhanced recurrent neural network model is first used to extract the shape The spatial characteristics of As the dimension parameter, then the shape is The spatial features are flattened into a one-dimensional vector, and then the fully connected layer FC is used to compress the dimension to a uniform dimension again. The shape is The spatial features of the global enhanced recurrent neural network model are extracted by the position encoding module using a two-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the two-dimensional sinusoidal position encoding PE or the learnable position encoding PE is embedded with the position encoding of the electrode channel of the input data in the preset electrode channel position two-dimensional table; the features output by the encoder layer are obtained by the recurrent neural network model to obtain the local correlation and global correlation between different units to obtain the shape of spatial characteristics.

[0015] Optionally, when the shallow features after fusion and dimensionality unification based on global attention and recurrent neural network are sequentially subjected to spatial feature extraction, dimensionality compression and time feature extraction to obtain EEG spatiotemporal features, the time feature extraction includes: The spatial feature construction shape is The combined features of is the number of sample sub-segments, is the dimension parameter; the shape is The combined features are automatically extracted using a global enhanced recurrent neural network model The shape of the local and global associations between the sample sub-segments after global information enhancement is: The time characteristics of is a dimension parameter; and when the globally enhanced recurrent neural network model performs time feature extraction, the position encoding module adopts a one-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the one-dimensional sinusoidal position encoding PE or the learnable position encoding PE embeds the sequential encoding of the sample sub-segments in the original sample segments; the encoder layer is an encoder using a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement; the features output by the encoder layer are obtained through the recurrent neural network model to obtain the local and global associations between different units to obtain a shape of time characteristics.

[0016] Optionally, when classifying the EEG spatiotemporal features to obtain the EEG decoding results, the classifier used is a fully connected classification module, and the fully connected classification module is composed of two fully connected layers FC and an activation function.

[0017] In addition, the present invention also provides an EEG decoding system based on global attention and recurrent neural network, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network.

[0018] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network through a processor.

[0019] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network through a processor.

[0020] Compared with the prior art, the present invention can mainly achieve the following beneficial effects:

[0021] 1. The present invention introduces the multi-head self-attention mechanism in deep learning, integrates global correlation information into each unit in the recurrent neural network, and then uses the sequential calculation of the recurrent neural network to further extract local and global correlations, which can fully improve the network's EEG signal spatiotemporal feature extraction capability and decoding accuracy.

[0022] 2. The globally enhanced recurrent neural network module proposed in the present invention can be portably migrated to various other EEG signal decoding scenarios to extract spatial correlation information between different electrodes and temporal correlation information between different time nodes of samples, and is not limited to the EEG decoding method based on global attention and recurrent neural network proposed in the present invention.

[0023] 3. The EEG decoding method based on global attention and recurrent neural network proposed in the present invention can be used for a variety of EEG decoding downstream tasks, including fitting and classification. More specifically, it can also involve a variety of cognitive tasks, such as alertness assessment and emotion recognition, and has strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0025] Figure 2 Schematic diagram of the principle of sample division in an embodiment of the present invention.

[0026] Figure 3 Schematic diagram of the network structure for extracting shallow features and performing fusion and dimensionality unification in an embodiment of the present invention.

[0027] Figure 4 Schematic diagram of the network structure of the globally reinforced recurrent neural network model in an embodiment of the present invention.

[0028] Figure 5 Schematic diagram of the electrode channel position distribution in an embodiment of the present invention.

[0029] Figure 6 Schematic diagram of the principle of position coding in an embodiment of the present invention.

[0030] Figure 7 Schematic diagram of the network structure for spatial feature extraction and dimensionality compression in an embodiment of the present invention.

[0031] Figure 8 Schematic diagram of the network structure for time feature extraction in an embodiment of the present invention.

[0032] Figure 9 Schematic diagram of the network structure for classifying EEG spatiotemporal features in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention aims to integrate global correlation information into each unit of a recurrent neural network by introducing a multi-head self-attention mechanism in deep learning, and then further extract local and global correlations by using the sequential calculation of the recurrent neural network, which can fully improve the network's ability to extract spatiotemporal features of EEG signals and decoding accuracy. In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] like Figure 1 As shown, the EEG decoding method based on global attention and recurrent neural network in this embodiment includes the following steps:

[0035] S1, divide the original EEG signal segment to be decoded into sample sub-segments;

[0036] S2, extract shallow features from sample sub-segments;

[0037] S3, fuses shallow features and unifies dimensions;

[0038] S4, based on global attention and recurrent neural network, the shallow features after fusion and dimension unification are sequentially subjected to spatial feature extraction, dimension compression and temporal feature extraction to obtain EEG spatiotemporal features;

[0039] S5, classify the EEG spatiotemporal features to obtain the EEG decoding results.

[0040] like Figure 2 As shown, in step S1 of this embodiment, the original EEG signal segments to be decoded are divided into samples, including: dividing the original EEG signal segments into samples according to the time length Divide into several sample segments, and then divide the sample segments into smaller preset durations The shape obtained by division is The sample sub-segment of is the number of electrode channels in the original EEG signal, is the downsampling rate of the EEG signal. For example, in this embodiment, In this embodiment, the sample segment is further divided into sample sub-segments according to the preset smaller duration of 0.5s. Each sample segment contains Sample sub-segments, the dataset used , ,The original EEG signal segment to be decoded contains a total of 1700 sample sub-segments.

[0041] Extracting shallow features from sample sub-segments can use the required feature extraction methods, including but not limited to manual extraction features such as differential entropy and power spectral density, as well as features extracted based on deep learning models such as one-dimensional convolution and pooling. For example, as an optional implementation, Figure 3 As shown, in step S2 of this embodiment, extracting shallow features from the sample sub-segment includes: dividing the data of each electrode channel in the sample sub-segment into different frequency band windows, using a bandpass filter to filter based on the frequency window of a specified size to obtain a variety of different frequency band signals, and calculating the differential entropy feature (DE) of each frequency band signal to obtain a dimension of The differential entropy feature is used as the shallow feature extracted. In this embodiment, for the data of 1 to 51 Hz, for example, according to the traditional 5-band ( ) division method, every 5Hz or every 2Hz frequency window, each electrode will obtain 5-, 10-, and 25-dimensional differential entropy features respectively.

[0042] like Figure 3 As shown, in step S3 of this embodiment, the shallow features are fused and dimensionally unified, including: the shallow feature input is divided into as well as The fusion and dimension unification module composed of two fully connected layers FC is used to unify the shallow feature shapes into ,in is the number of electrode channels in the original EEG signal, is the dimension parameter, and the activation function used between the two fully connected layers FC is the linear rectifier function Relu. In the two fully connected layers FC of the fusion and dimension unification module, the dimension of the first fully connected layer is half of the second fully connected layer, and the feature dimension will be unified to the dimension of the second fully connected layer. Represented as follows, the data shape of each sample sub-segment is unified as In this embodiment Set to 64.

[0043] like Figure 4 As shown, in step S4 of this embodiment, both spatial feature extraction and temporal feature extraction are implemented by using a globally enhanced recurrent neural network model. The globally enhanced recurrent neural network model includes a multi-level cascade unit (specifically 2 levels in this embodiment, which can be selected as needed). The cascade unit is composed of a position encoding module, an encoder layer, and a recurrent neural network model connected in sequence. The encoder layer is an encoder using a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement. When performing spatial feature extraction, the position encoding module embeds the position encoding of the electrode channel of the input data in the preset two-dimensional table of electrode channel positions. When performing temporal feature extraction, the position encoding module embeds the sequential encoding of the sample sub-segment in the original sample segment. Among them, the cascade unit is composed of a position encoding module, an encoder layer, and a recurrent neural network model connected in sequence. The position encoding is directly added to the input signal, and then imported into the encoder layer of the transformer model to obtain global information, and finally imported into the recurrent neural network to further obtain local and global information. As shown Figure 4 As shown, Indicates the number of input vectors when extracting spatial features , represents the number of electrodes, when extracting time features , which indicates the number of sample sub-segments in the sample segment, that is, the number of time periods of the sample. for or .

[0044] The position encoding module is used to add inherent positional information to the input data. When extracting spatial features, the position encoding module embeds the positional codes of the input data's electrode channels within a pre-set two-dimensional table of electrode channel positions. This table can be used to characterize the orientation and distance relationships between electrode points. When extracting temporal features, the position encoding module embeds the sequence codes of the sample subsegments within the original sample segments. The electrode channel sequence table can be used to characterize the order and distance relationships between time nodes. In addition to the aforementioned encoding methods, other reasonable position encoding methods that can characterize temporal order and spatial orientation information can also be used. Figure 5 is the electrode channel position, wherein the dotted box is the electrode channel position area used in this embodiment; Figure 6 It is a two-dimensional table of electrode channel positions. The numbers in brackets at the bottom of each electrode in the two-dimensional table of electrode channel positions are the order codes of the electrodes in the electrode channel sequence table. The position encoding module is used to increase the inherent position information between the input data. In this embodiment, the one-dimensional sinusoidal PE is used to characterize the order and distance relationship between time nodes for time feature extraction; the two-dimensional sinusoidal PE and the learnable PE are used to characterize the orientation and distance relationship between electrode points for spatial feature extraction. Among them, the two-dimensional sinusoidal PE needs to first map the image into a two-dimensional image, such as Figure 6 As shown, the electrodes are then encoded according to their positions in the image; for one-dimensional sinusoidal PE, the electrodes only need to be arranged in one dimension, as shown Figure 6 The numbers in parentheses on the right indicate the order in which the calculations are performed. For learnable PE, simply add the learnable parameters to the original vector, allowing the network to automatically learn the position information. The calculation formula for one-dimensional sinusoidal PE is a well-known method. Two-dimensional sinusoidal PE and learnable PE are also well-known methods. In this embodiment, two-dimensional sinusoidal PE is preferred for spatial feature extraction, or it is not added; one-dimensional sinusoidal PE is preferred for temporal feature extraction.

[0045] In this embodiment, the encoder layer uses a transformer encoder model (TranEnlayer), which is used to capture global information between input data. TranEnlayer consists of a multi-head self-attention mechanism and a feedforward layer network. The multi-head self-attention mechanism captures global information between input vectors, while the feedforward layer expands the linear and nonlinear feature representations of this global information. Encoder layers can be stacked or used individually. The data output by the encoder layer has the same dimensionality as the input data.

[0046] The recurrent neural network model is used for sequential computation to further obtain the local and global correlations between different nodes of the recurrent neural network. It includes traditional recurrent neural networks and variants of recurrent neural networks with the same sequential computation characteristics, such as the Long-Short-Term Memory network (LSTM). The dimension of the recurrent neural network output will become the dimension of the hidden layer set by the recurrent neural network, and the dimension of the hidden layer will be used. As an optional implementation, the recurrent neural network model in this embodiment specifically adopts a long short-term memory network (LSTM). It is composed of multiple chain-like repeating units, which can be calculated sequentially to obtain local and global correlations between different units. The dimension of the LSTM output will become the set hidden layer dimension, and Indicates. Figure 4 As shown, In the first globally reinforced LSTM unit, , in the second globally reinforced LSTM unit is In this embodiment Set to 32.

[0047] After the global enhanced recurrent neural network model is built, the spatiotemporal signal extraction of EEG signals can be performed based on the global enhanced recurrent neural network model, including spatial feature extraction, dimension compression and temporal feature extraction of EEG signals. Figure 7 As shown, in this embodiment, the spatial feature extraction and dimension compression include: for the shape The shallow features of is the number of electrode channels in the original EEG signal, As the dimension parameter, the global enhanced recurrent neural network model is first used to extract the shape The spatial characteristics of As the dimension parameter, then the shape is The spatial features are flattened into a one-dimensional vector, and then the fully connected layer FC is used to compress the dimension to a uniform dimension again. The shape is The spatial features of the global enhanced recurrent neural network model are extracted by the position encoding module using a two-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the two-dimensional sinusoidal position encoding PE or the learnable position encoding PE is embedded with the position encoding of the electrode channel of the input data in the preset electrode channel position two-dimensional table; the features output by the encoder layer are obtained by the recurrent neural network model to obtain the local correlation and global correlation between different units to obtain the shape of In this embodiment, the dimension compression includes a fully connected layer, which expands the high-dimensional features of the sample sub-segment into a one-dimensional vector, and then uses the fully connected layer to compress the dimension and compress it again to a unified dimension. Each contains The shape of the sample segment of sample sub-segments becomes .

[0048] like Figure 8 As shown, in this embodiment, the time feature extraction includes: The spatial feature construction shape is The combined features of is the number of sample sub-segments, is the dimension parameter; the shape is The combined features are automatically extracted using a global enhanced recurrent neural network model The shape of the local and global associations between the sample sub-segments after global information enhancement is: The time characteristics of is a dimension parameter; and when the globally enhanced recurrent neural network model performs time feature extraction, the position encoding module adopts a one-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the one-dimensional sinusoidal position encoding PE or the learnable position encoding PE embeds the sequential encoding of the sample sub-segments in the original sample segments; the encoder layer is an encoder using a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement; the features output by the encoder layer are obtained through the recurrent neural network model to obtain the local and global associations between different units to obtain a shape of In this embodiment, when extracting time features, the data of each sample segment , imported into the global enhanced recurrent neural network module, the global enhanced recurrent neural network module will automatically extract The local and global temporal correlations between the sample sub-segments after global information enhancement form new features that incorporate deep temporal features. .

[0049] like Figure 9 As shown, in this embodiment, when classifying EEG spatiotemporal features to obtain EEG decoding results, the classifier used is a fully connected classification module. The fully connected classification module consists of two fully connected layers (FCs) and an activation function. The fully connected layers are used to expand high-dimensional features and then reduce their dimensionality. Optional downstream tasks include fitting and classification tasks. In the fitting task, the dimension of the last fully connected layer is one, no activation function is required, and the loss is calculated using the mean squared error loss function and returned. In the classification task, the dimension of the last fully connected layer is the specified number of categories, and the loss is calculated using the cross-entropy loss function using a normalized exponential function and returned. In this embodiment, the alertness estimation fitting task is specifically performed. The original EEG signal segment is the EEG signal collected by the driver while driving. The EEG decoding result is the PERCLOS value indicating the driver's alert or fatigued driving state, respectively. After fitting, EEG samples with a PERCLOS value greater than 0.35 are classified as fatigued, and EEG samples with a PERCLOS value less than 0.35 are classified as alert, and further classification tasks are performed. The PERCLOS value is a well-known alertness label in the field of alertness estimation. In this embodiment, the core downstream task is fitting. The dimension of the last fully connected layer is 1, and no activation function is required. The loss is calculated using the mean squared error loss function and then returned. The dimension of the penultimate fully connected layer is set to 120 in this embodiment.

[0050] To verify the effectiveness of the EEG decoding method based on global attention and recurrent neural networks in this embodiment, an alertness estimation experiment was conducted using the publicly available SEED-VIG dataset. SEED-VIG is a subset of the SEED dataset, provided by the Brain-Inspired Computing and Machine Intelligence (BCMI) Laboratory of Shanghai Jiao Tong University. The experiment was conducted on a virtual driving system. The simulation scenario was designed as a straight, monotonous four-lane road. Participants were required to drive straight along this four-lane road. The simulated driving experiment lasted two hours, after lunch, a time when fatigue is most likely to be induced. The dataset used only a small number of electrodes in the occipital and temporal lobes. Excluding the reference electrode, the dataset contains data from 17 electrode channels, all arranged in a 10-20 electrode system. The data from each electrode was downsampled to 200 Hz. The dataset contains data from 23 participants. During the experiment, all participant data from the SEED-VIG dataset were pooled and the algorithm performance was tested using five-fold cross-validation. Specifically, the dataset was randomly divided into five folds, one of which served as the test set and the remaining four as the training set. The training process lasted for 300 epochs. Each fold experiment was repeated five times, and the best result from each experiment was recorded. The final model performance was determined by the mean and variance of the five-fold results. To minimize the impact of model initialization, five different initialization seeds were used, and the average result was taken as the final model performance. All experimental results were evaluated using five classification metrics (accuracy, precision, recall, F1 score, and kappa coefficient) and two regression metrics (root mean square error (RMSE) and correlation coefficient). Precision measures the proportion of predicted positive classes that are true positives, recall measures the proportion of true positives that are correctly identified, and the F1 score combines the two and is applicable to class imbalance. The kappa coefficient measures the consistency of predictions and accounts for randomness. The root mean square error reflects the deviation between the predicted and actual values, and the correlation coefficient measures the linear correlation between the predicted and actual values. Together, these metrics provide a comprehensive assessment of classification and regression performance.

[0051] The EEG decoding method based on global attention and recurrent neural network in this embodiment is compared with the currently available advanced algorithms, including SFT-Net, EEG-Comformer, TSception, EEGNet, DeepConvNet, ShallowConvNet, AMS-CNN, EEG-Conv, EEG-Conv-R and ESTCNN. The results of the method in this embodiment and other existing methods are finally obtained as shown in Table 1.

[0052] Table 1 Comparison of the results of the method in this embodiment with other existing methods

[0053]

[0054] As shown in Table 1, except for the recall rate which is slightly lower than that of SFT-Net and ESTCNN, the other indicators of the method in this embodiment are better than those of other existing methods. The comparison results prove the effectiveness of the method in this embodiment.

[0055] At the same time, in order to verify the effectiveness of the proposed global enhanced LSTM module, this embodiment designed a series of ablation experiments, and the evaluation results are shown in Table 2.

[0056] Table 2 Ablation experimental study on the global reinforcement recurrent neural network model

[0057]

[0058] As shown in Table 2, the complete module performs best across all metrics. Ablation of either the LSTM or TranEnlayer module results in a decrease in all metrics. Ablation of the entire module results in a significant drop in all metrics, with accuracy dropping from 0.886 to 0.763, the Kappa coefficient dropping from 0.751 to 0.479, the minimum root mean square error improving from 0.100 to 0.171, and the maximum correlation coefficient dropping from 0.929 to 0.774. This demonstrates the effectiveness of the TranEnlayer and LSTM combination in the proposed module; ablating either module significantly degrades performance.

[0059] In summary, the EEG decoding method based on global attention and recurrent neural networks in this embodiment includes sample partitioning, shallow feature extraction, shallow feature fusion and dimensionality unification, a globally enhanced recurrent neural network module for extracting EEG spatiotemporal features, and a fully connected downstream task output module. The globally enhanced recurrent neural network module includes a position encoding, a transformer encoder layer (including a multi-head self-attention mechanism), and a recurrent neural network. The globally enhanced recurrent neural network module extracts EEG spatiotemporal features, including spatial feature extraction, dimensionality compression, and temporal feature extraction for EEG signals. This embodiment method incorporates global correlation information into each unit of the recurrent neural network by introducing the multi-head self-attention mechanism in deep learning. Then, the sequential calculation of the recurrent neural network is used to further extract local and global correlations, which can fully improve the network's EEG signal spatiotemporal feature extraction capability and decoding accuracy. This embodiment method can integrate multi-head self-attention global modeling with recurrent neural network temporal characteristics to improve the accuracy and robustness of EEG signal spatiotemporal feature decoding, and is suitable for EEG decoding for various cognitive tasks such as alertness estimation and emotion recognition.

[0060] In addition, this embodiment also provides an EEG decoding system based on global attention and recurrent neural network, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network.

[0061] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network through a processor.

[0062] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network through a processor.

[0063] Those skilled in the art should understand that the technical solution provided by the present invention may be in the form of a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An EEG decoding method based on global attention and recurrent neural network, characterized in that: The steps include: Divide the original EEG signal segment to be decoded into sample sub-segments; Extract shallow features from sample sub-segments; Fusion of shallow features and unification of dimensions; Based on global attention and recurrent neural network, the shallow features after fusion and dimension unification are sequentially subjected to spatial feature extraction, dimension compression and temporal feature extraction to obtain EEG spatiotemporal features; Wherein both spatial feature extraction and temporal feature extraction are implemented by adopting a globally enhanced recurrent neural network model, wherein the globally enhanced recurrent neural network model includes a multi-stage cascade unit, wherein the cascade unit is composed of a position encoding module, an encoder layer and a recurrent neural network model connected in sequence, wherein the encoder layer is an encoder adopting a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement, and when performing spatial feature extraction, the position encoding module embeds the position encoding of the electrode channel of the input data in a preset electrode channel position two-dimensional table, and when performing temporal feature extraction, the position encoding module embeds the sequential encoding of the sample sub-segment in the original sample segment; Classify EEG spatiotemporal features to obtain EEG decoding results; The method of extracting shallow features from the sample sub-segment includes: dividing the data of each electrode channel in the sample sub-segment into different frequency band windows, using a bandpass filter to filter based on the frequency window of a specified size to obtain a variety of different frequency band signals, and calculating the differential entropy feature of each frequency band signal to obtain a dimension of The differential entropy features are used as the extracted shallow features.

2. The EEG decoding method based on global attention and recurrent neural network according to claim 1, characterized in that: The sample division of the original EEG signal segments to be decoded includes: dividing the original EEG signal segments into samples according to the time length Divide into several sample segments, and then divide the sample segments into smaller preset durations The shape obtained by division is The sample sub-segment of is the number of electrode channels in the original EEG signal, is the downsampling rate of the EEG signal.

3. The EEG decoding method based on global attention and recurrent neural network according to claim 1, characterized in that: The fusion and dimension unification of shallow features include: inputting shallow features with dimensions of as well as The fusion and dimension unification module composed of two fully connected layers FC is used to unify the shallow feature shapes into ,in is the number of electrode channels in the original EEG signal, is a dimension parameter, and the activation function used between the two fully connected layers FC is the linear rectification function Relu.

4. The EEG decoding method based on global attention and recurrent neural network according to claim 1, characterized in that: The spatial feature extraction, dimensionality compression and time feature extraction of the shallow features after fusion and dimensionality unification based on global attention and recurrent neural network are performed in sequence to obtain EEG spatiotemporal features. The spatial feature extraction and dimensionality compression include: for the shape of The shallow features of is the number of electrode channels in the original EEG signal, As the dimension parameter, the global enhanced recurrent neural network model is first used to extract the shape The spatial characteristics of As the dimension parameter, then the shape is The spatial features are flattened into a one-dimensional vector, and then the fully connected layer FC is used to compress the dimension to a uniform dimension again. The shape is The spatial features of the global enhanced recurrent neural network model are extracted by the position encoding module using a two-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the two-dimensional sinusoidal position encoding PE or the learnable position encoding PE is embedded with the position encoding of the electrode channel of the input data in the preset electrode channel position two-dimensional table; the features output by the encoder layer are obtained by the recurrent neural network model to obtain the local correlation and global correlation between different units to obtain the shape of spatial characteristics.

5. The EEG decoding method based on global attention and recurrent neural network according to claim 1, characterized in that: When the shallow features after fusion and dimension unification are sequentially subjected to spatial feature extraction, dimension compression and time feature extraction based on global attention and recurrent neural network to obtain EEG spatiotemporal features, the temporal feature extraction includes: The spatial feature construction shape is The combined features of is the number of sample sub-segments, is the dimension parameter; the shape is The combined features are automatically extracted using a global enhanced recurrent neural network model The shape of the local and global associations between the sample sub-segments after global information enhancement is: The time characteristics of is a dimension parameter; and when the globally enhanced recurrent neural network model performs time feature extraction, the position encoding module adopts a one-dimensional sinusoidal position encoding PE or a learnable position encoding PE, and the one-dimensional sinusoidal position encoding PE or the learnable position encoding PE embeds the sequential encoding of the sample sub-segments in the original sample segments; the encoder layer is an encoder using a transformer model to utilize the multi-head self-attention mechanism of the transformer model to achieve global attention enhancement; the features output by the encoder layer are obtained through the recurrent neural network model to obtain the local and global associations between different units to obtain a shape of time characteristics.

6. The EEG decoding method based on global attention and recurrent neural network according to claim 1, characterized in that: When classifying the EEG spatiotemporal features to obtain the EEG decoding results, the classifier used is a fully connected classification module, which is composed of two fully connected layers FC and an activation function.

7. An EEG decoding system based on global attention and recurrent neural network, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network as described in any one of claims 1 to 6 through a processor.

9. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the EEG decoding method based on global attention and recurrent neural network as described in any one of claims 1 to 6 through a processor.

Images