Transform-based electroencephalogram recognition method
Through the Transformer-based EEG recognition method, the global spatial and timing characteristics of EEG signals are extracted, and the problem of incomplete extraction of space-time information in the prior art is solved, thereby achieving high-accurate EEG signal recognition.
Patent Information
- Application Number
- CN202411936262.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-05-09
AI Technical Summary
The prior art has incomplete extraction of spatiotemporal information during the extraction of EEG signal features, resulting in low recognition performance.
Using the Transformer-based EEG recognition method, global spatial features are extracted through the brain area Transformer module, and the timing Transformer module extracts global timing features, and combined with the space-time multi-scale convolution module, deep-level spatiotemporal features are obtained, and finally the two-layer fully connected neural network and Softmax layer are identified and classified.
Effectively capture the global spatial and timing characteristics of EEG signals, suppress category offset problems, significantly reduce the error detection rate, and improve the recognition accuracy of EEG signals.
Smart Images
Figure CN119961780A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of radar signal processing, and in particular relates to a Transformer-based electroencephalogram recognition method. Background Art
[0002] Stereoscopic vision is an important physiological indicator of visual function. It is the key to good motion control and accurate stereoscopic cognition. From a medical perspective, stereoscopic vision is reflected in stereoscopic visual acuity, that is, the minimum disparity that triggers stereoscopic perception. Early stereoscopic visual acuity measurement methods include: Frisby distance test and Howard-Dolman stereoscopic vision test. However, the above methods have monocular cues, resulting in low test sensitivity and certain limitations. The presentation of random-dot stereograms (RDS) corresponds to both eyes and can completely eliminate the influence of monocular cues, so RDS has gradually been developed in the field of stereoscopic visual cognition. Methods based on RDS are applied to the test of stereoscopic visual acuity. Among them, the TNO test based on red and green RDS proposed by Kathleen et al. is commonly used. In recent years, studies have found that people are more sensitive to dynamic random-dot stereograms (DRDS), and stereoscopic vision research using DRDS can achieve higher sensitivity. Based on DRDS, Brian et al. studied the influence of two cues, motion time and motion speed difference, on binocular disparity during motion perception. Shen et al. first used the DRDS graded test of stereo visual acuity threshold for horizontal motion stereo vision research. In addition, stereo vision research can also use sophisticated acquisition equipment to obtain various physiological electrical signals of the subjects, including: electrooculographic signals, electromyographic signals, electrocardiographic signals, electroencephalography (EEG) signals, etc., and then design signal feature extraction methods to further analyze and obtain more general and objective results. Among the above indicators, EEG, as an accurate and high-temporal resolution technology, is widely used in the field of stereo vision recognition.
[0003] In recent years, deep learning algorithms can automatically obtain deeper intrinsic feature representations from raw data and are gradually applied to EEG signal classification tasks. Zhang et al. proposed a spatial–temporal recurrent neural network (STRNN) to combine spatiotemporal EEG information for emotion recognition. Li et al. proposed an attention-based multi-scale fusion CNN for MI EEG signal decoding. The algorithm uses a one-dimensional spatiotemporal CNN to learn spatiotemporal information and introduces an attention mechanism to make EEG features easier to distinguish. Chen et al. converted the original EEG sequence into a grid representation based on the spatial positions of different electrodes, and then used a two-dimensional CNN to obtain its spatiotemporal features. Although the above networks have advantages in local feature extraction, they cannot learn the global spatial relationship between EEG electrodes. Neuroimaging studies have found that visual information processing is responsible for visual pathways across multiple brain regions. Therefore, some papers point out that the complex spatial relationship between different EEG electrodes is of great significance for EEG recognition and classification tasks. Summary of the invention
[0004] The purpose of the present invention is to provide a Transformer-based EEG recognition method to solve the problems of incomplete spatiotemporal information extraction and poor recognition performance in the feature extraction process.
[0005] To achieve the above object, the present invention adopts the following technical solution:
[0006] On the one hand, this specification provides a Transformer-based EEG recognition method, comprising:
[0007] Step 102, preprocessing the acquired EEG signal to obtain shallow features;
[0008] Step 104, inputting the shallow features into the brain region Transformer module, extracting the global spatial features, and obtaining the first features;
[0009] Step 106, input the first feature into the time series Transformer module, extract the global time series feature, and obtain the second feature;
[0010] Step 108, input the second feature into the temporal Transformer module, extract the deep spatiotemporal features, and obtain the third feature;
[0011] Step 110, using a two-layer fully connected neural network and a Softmax layer to identify and classify the third feature to obtain a prediction result.
[0012] On the other hand, this specification provides a Transformer-based EEG recognition device, comprising:
[0013] The EEG signal preprocessing module is used to preprocess the acquired EEG signals to obtain shallow features;
[0014] The spatial feature extraction module is used to input the shallow features into the brain region Transformer module, extract the global spatial features, and obtain the first feature;
[0015] A time feature extraction module, used for inputting the first feature into a time series transformer module, extracting a global time series feature, and obtaining a second feature;
[0016] A deep feature extraction module is used to input the second feature into the temporal transformer module, extract the deep spatiotemporal feature, and obtain the third feature;
[0017] The feature recognition and classification module is used to identify and classify the third feature using a two-layer fully connected neural network and a Softmax layer to obtain a prediction result.
[0018] Based on the above technical solution, this specification can achieve the following technical effects:
[0019] This method uses the brain region Transformer module to capture EEG local and global spatial features from within the brain region and the entire brain region, and simultaneously learns the spatial connections between electrodes in different brain regions and on the entire brain region; uses the time Transformer module to fully extract the global temporal dependencies within different time segments, effectively retains the long-distance temporal signal characteristics, and uses the spatiotemporal multi-scale convolution module to further learn the temporal and spatial features to obtain advanced EEG spatiotemporal features, effectively suppressing the category shift problem caused by multi-category sample data, greatly reducing the false detection rate, and accurately identifying EEG signals, thereby solving the problems of incomplete spatiotemporal information extraction and poor recognition performance in the feature extraction process. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 The figure is a flowchart of a Transformer-based EEG recognition method in one embodiment of the present invention.
[0021] Figure 2 This is a diagram of the brain area structure division in one embodiment of the present invention.
[0022] Figure 3 This is a brain region Encoder module in one embodiment of the present invention.
[0023] Figure 4 It is a timing encoder module in an embodiment of the present invention.
[0024] Figure 5The figure is a schematic diagram of the structure of a Transformer-based EEG recognition device in one embodiment of the present invention.
[0025] Figure 6 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION
[0026] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.
[0027] It should be noted that, in order to clearly explain the content of the present invention, the present invention specifically cites multiple embodiments to further illustrate different implementations of the present invention, wherein the multiple embodiments are enumerated rather than exhaustive. In addition, for the sake of brevity of explanation, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, the contents not mentioned in the subsequent embodiments can refer to the previous embodiments accordingly.
[0028] Example 1
[0029] Please refer to Figure 1 , Figure 1 The figure shows a Transformer-based EEG recognition method provided in this embodiment. In this embodiment, the method includes:
[0030] Step 102, preprocessing the acquired EEG signal to obtain shallow features;
[0031] In this embodiment, one implementation of step 102 is:
[0032] Step 202, quantizing and encoding the acquired EEG signal of the subject and converting it into a continuous digital signal;
[0033] Step 204, down-sampling the digital signal to obtain a digital sampling signal;
[0034] Step 206, performing bandpass filtering, independent component analysis and overlapping slicing processing on the digital sampling signal in sequence to obtain a two-dimensional matrix numerical signal;
[0035] Step 208, performing one-dimensional convolution operation and reshaping operation on the two-dimensional matrix numerical signal to obtain shallow features.
[0036] Specifically, the above preprocessing method is as follows:
[0037] After collecting the EEG signals of the experimental subjects, the quantized codes are converted into continuous digital signals, the digital signals are downsampled, and bandpass filtering, independent component analysis and overlapping slicing are performed in sequence to convert the digital signals into two-dimensional matrix numerical signals, which are specifically explained by the following examples:
[0038] The digital signals were band-pass filtered to remove background noise, the data baseline was removed by the EEGLab toolbox in Matlab, and artifacts were removed by independent component analysis.
[0039] All the above digital signals are repeatedly sampled at 256 Hz to eliminate the redundancy of the above digital signals.
[0040] By converting the above digital signal into a series of 1s samples through overlapping signal slicing, the data set can be effectively expanded to enhance the learning ability of the neural network.
[0041] In other embodiments, the length of the sample in the time dimension may be adjusted according to actual conditions.
[0042] In this example, stereogram recognition dataset A and stereogram recognition dataset B are taken as examples, namely SRDA and SRDB. The basic formats of the above datasets are shown in Table 1.
[0043] Table 1 Basic format of the dataset
[0044]
[0045] Among them, subjects refers to the number of subjects, trials refers to the number of trials, electrodes refers to the number of EEG electrodes, and sampling points refers to the number of sampling points of the EEG signal of each electrode channel in a single trial.
[0046] Step 104, inputting the shallow features into the brain region Transformer module, extracting the global spatial features, and obtaining the first features;
[0047] In this embodiment, one implementation of step 104 is:
[0048] Step 302, based on the distribution of brain regions and the position distribution of EEG electrodes, the shallow features are divided; the shallow features are signals of several electrodes;
[0049] Step 304, the divided shallow features are respectively sent to the brain region Transformer modules of the corresponding brain regions to learn the intra-regional spatial features of the brain regions;
[0050] Step 306, combining the intra-regional spatial features of different brain regions to obtain the intra-regional spatial features of the entire brain;
[0051] Step 308, the global spatial features of the entire brain region are sent to the brain region Transformer module to learn the spatial features of the entire brain region and obtain the global spatial features.
[0052] In this embodiment, the brain area distribution includes 6 areas, namely the frontal area, left temporal area, right temporal area, central area, parietal area, and occipital area; the position distribution of the EEG electrodes is 7 electrodes in the frontal area, 5 electrodes in the left temporal area, 5 electrodes in the right temporal area, 5 electrodes in the central area, 5 electrodes in the parietal area, and 3 electrodes in the occipital area, a total of 30 electrodes.
[0053] Specifically, the EEG signal is used as an input signal, and a brain region Transformer module is used to extract a first feature, wherein the first feature at least includes a global spatial feature of the EEG signal in each brain region and on the entire brain region, and a global spatial dependency between all electrodes;
[0054] Specifically, a specific implementation of step 104 is as follows:
[0055] Step 122: Divide the EEG signal into brain regions based on the EEG electrode positions. Combined with prior knowledge such as brain region distribution and EEG electrode position distribution, the specific distribution of the 30 EEG electrodes in the model of this paper is shown in the following table:
[0056] Table 2 EEG electrode distribution table
[0057]
[0058]
[0059] Mainly based on the structure of the human brain, it is divided into 6 parts, including 7 electrodes in the frontal area, 5 electrodes in the left temporal area, 5 electrodes in the right temporal area, 5 electrodes in the central area, 5 electrodes in the parietal area, and 3 electrodes in the occipital area. The distribution of the electrodes in the brain is as follows Figure 2 shown.
[0060] Step 124: Send the divided EEG signals to the corresponding brain region Transformer Encoder module to learn the global information in the brain region. Then, the features from different brain regions are spliced to obtain the feature set of all EEG electrodes on the entire brain.
[0061] Step 126: Then, the signals of all electrodes are sent to the brain region Transformer Encoder module to learn the global importance information of each electrode in the entire brain region. After the above operation, the brain region Transformer module extracts the global importance information in each brain region and on the entire brain region.
[0062] In this embodiment, the main processes involved in the above implementation are as follows:
[0063] Assume that the input EEG signal is X = [x1, ..., x N ]∈R N×L , x N ∈R L , where the number of electrodes is N and the length of the time series is P. First, without destroying the spatiotemporal information of the signal, a time convolution and reshaping operation with a size of 1×7 and a step size of 1×2 is used to convert the signal into feature X*. The calculation formula is as follows:
[0064]
[0065] Then the signals of N electrodes are divided into 6 brain regions and sent to the brain region encoder module respectively, such as Figure 3 shown.
[0066] The brain region encoder in the figure above mainly adopts the Transformer Encoder structure. In this module, the feature learning process in each brain region does not involve dimensionality changes. The calculation process of the i-th brain region is shown below.
[0067]
[0068] Where i = 1, 2, ..., 6. Then the features of all brain regions are merged and sent to the Encoder module of the entire brain region to obtain the output feature Z of the Transformer module of the brain region. The calculation formula is as follows:
[0069]
[0070] After passing through the brain region Transformer module, the network learns the spatial information within each brain region and the entire brain region. The feature map learned by this module is further sent to the timing Transformer module to extract EEG timing information.
[0071] Step 106, input the first feature into the time series Transformer module, extract the global time series feature, and obtain the second feature;
[0072] In this embodiment, one implementation of step 106 is:
[0073] Step 402, transposing the first feature to obtain a transposed feature;
[0074] Step 404: input the transposed features into the time series Transformer module for global weighted learning, extract the global time series features, and obtain the second features.
[0075] In this embodiment, after step 404, the following steps are further included:
[0076] The second feature is dimensionally expanded and reshaped into a three-dimensional second feature.
[0077] Specifically, the first feature is used as an input signal, and a time Transformer module is used to extract a second feature, wherein the second feature at least includes a global temporal feature of the EEG signal;
[0078] In this embodiment, a specific implementation of step 106 is as follows:
[0079] Step 132, transposing the features learned by the brain region Transformer module in TER-TSAN as input;
[0080] Step 134: Perform global weighted learning on the input using a temporal Transformer module to obtain an output feature S;
[0081] Step 136: Dimensionally expand the output feature and reshape it into a three-dimensional feature S*.
[0082] In this embodiment, the main processes involved in the above implementation are as follows:
[0083] For the output feature Z∈R of the Transformer module in the brain region N×P , first transpose it, and then use the Transformer Encoder structure to learn the global weighted features of P time slice sequences of length 1×N. The time series Encoder module is as follows Figure 4 shown.
[0084] In the temporal Transformer module, the learned feature S is implemented as follows:
[0085] S=encoder(permute(Z)),S∈R P×N
[0086] After the temporal Transformer module, in order to ensure that the features in the network can participate in the subsequent convolutional feature learning, the feature map S needs to be dimensionally expanded and reshaped into a three-dimensional feature S*. The specific implementation is as follows.
[0087] S*=reshape(permute(S)),S*∈R 1×N×P
[0088] Step 108, input the second feature into the temporal Transformer module, extract the deep spatiotemporal features, and obtain the third feature;
[0089] In this embodiment, one implementation of step 108 is:
[0090] Step 502, inputting the second feature into three spatial multi-scale convolutional layers, respectively learning the deep spatial information of the EEG electrode signal from the local and overall perspectives, and obtaining deep spatial features of three scales;
[0091] Step 504, spatially parallel splicing the deep spatial features of the three scales to obtain high-level spatial features;
[0092] In step 506, the high-level spatial features are input into three time convolution layers of different scales to learn and fuse the high-level time information to obtain high-level spatiotemporal features, i.e., the third features.
[0093] In this embodiment, after step 506, the method further includes: applying 1×1 convolution and 4×4 pooling operations to the third feature to obtain a final fused feature.
[0094] Specifically, the spatiotemporal features are used as input. In order to reduce the number of features and speed up network training, 1×1 convolution and 4×4 pooling operations are used to obtain the final fusion feature F*. The calculation formula is as follows:
[0095] F*=avg_poll 4×4 (conv 1×1 (F)), F*∈R 16×15×32
[0096] Specifically, the second feature is used as input, and a spatiotemporal multiscale convolution module is used to extract a third feature, i.e., a final feature, wherein the third feature at least includes a deep spatiotemporal feature of the EEG signal.
[0097] In this embodiment, a specific implementation of step 108 is as follows:
[0098] Step 142: Perform higher-level feature extraction on the features learned from the above brain regions and timing modules. First, send the feature S* to three spatial multi-scale convolutional layers to learn the deep spatial information of EEG from local and global perspectives.
[0099] Step 144, further explore the temporal characteristics of the above output features, use three different scales of temporal convolution learning to fuse high-level temporal information, and obtain the final high-level EEG spatiotemporal features.
[0100] In this embodiment, the conversion involved in the above implementation is specifically as follows:
[0101] The spatiotemporal multiscale module combines the features S*∈R learned by the above brain regions and temporal modules 1×N×PFor higher-level feature extraction, in this paper, N = 30, P = 128. In order to further learn EEG features, the feature S* is first sent to three spatial multi-scale convolutional layers to learn EEG deep spatial information from local and global perspectives, and then the information of the three scales is spliced and organized into high-level spatial EEG features, which are mainly implemented as follows.
[0102] S1=conv 30×1 (S*), S1∈R 32×16×128
[0103] s2=conv 15×1 (S*), S2∈R 32×16×128
[0104] S1=conv 3×1 (S*), S3∈R 32×28×128
[0105] T=concat{S1, S2, S3}, T∈R 32×60×128
[0106] Then, three different scales of temporal convolution learning are used to fuse high-level temporal information to obtain the final high-level EEG spatiotemporal features. The main implementation is as follows.
[0107] T1=conv 1×7 (T), T1∈R 32×60×61
[0108] T2=conv 1×31 (T), T2∈R 32×60×49
[0109] T3=conv 1×127 (T), T3∈R 32×60×18
[0110] F=concat{T1, T2, T3}, F∈R 32×60×128
[0111] Step 110, using a two-layer fully connected neural network and a Softmax layer to identify and classify the third feature to obtain a prediction result.
[0112] Specifically, the fused feature F* is sent to two fully connected layers and one Softmax layer to achieve three-classification of EEG signals.
[0113] In this embodiment, after step 110, the following steps are further included:
[0114] Step 602: Use a cross entropy loss function to evaluate the predicted result and the acquired true result to obtain an evaluation result.
[0115] Specifically, in the model, the cross entropy loss L is used to evaluate the true label y i and predicted labels Due to the inconsistency between , the model loss function is shown below.
[0116]
[0117] In the above formula, M is the number of samples, Θ is the set of all trainable parameters in the network, ||.||1 represents the l1 norm of the vector, and λ is a constant parameter. In the loss function, the regularization term λ||Θ||1 is used to prevent overfitting.
[0118] In the above evaluation process, the SGD optimizer is used to optimize the trainable parameters in the model through back propagation, the learning rate is set to 0.01, the batch size is set to 16, and λ and μ are set to 0.00001 respectively.
[0119] In this embodiment, the performance of the method proposed in this embodiment is verified using stereogram recognition dataset A and stereogram recognition dataset B, namely SRDA and SRDB, to obtain prediction results. The accuracy after comparing the prediction results with the actual results is shown in Table 3.
[0120] Table 3 Experimental results of different methods on SRDA and SRDB datasets
[0121]
[0122] The parameters in the above verification method are the specific parameters proposed in this embodiment. The above method uses PyTorch1.8 to train on a server equipped with Intel CPU (i9-10900kF) and NVIDIA GPU (RTX 3090). After 100 iterations of each training, the best experimental results are saved, and the performance of the model (that is, the above method) is comprehensively evaluated using five-fold cross validation.
[0123] It can be seen that the average accuracy of the TER-TSAN EEG recognition method proposed in this embodiment on SRDA and SRDB is 96.84% and 96.33% respectively. Compared with other methods, its accuracy is at least 3% higher. Therefore, the feature extraction method and feature fusion method proposed in this embodiment can effectively improve the recognition accuracy of EEG signals, which can also prove the effectiveness of the above method.
[0124] This method targets two-dimensional matrix numerical EEG signals and designs a Transformer-based brain region timing analysis network based on the Transformer Encoder module. The network has excellent global timing and spatial information capture performance, effectively suppresses the category shift problem caused by multi-category sample data, greatly reduces the false detection rate, and improves the accuracy of EEG signal recognition.
[0125] In summary, this method uses the brain region Transformer module to capture EEG local and global spatial features from within the brain region and the overall brain region, and simultaneously learns the spatial connections between electrodes in different brain regions and on the overall brain region; uses the time Transformer module to fully extract the global temporal dependencies within different time segments, effectively retains the long-distance temporal signal characteristics, and uses the spatiotemporal multi-scale convolution module to further learn the temporal and spatial features to obtain advanced EEG spatiotemporal features, effectively suppressing the category shift problem caused by multi-category sample data, greatly reducing the false detection rate, and can accurately identify EEG signals, thereby solving the problems of incomplete spatiotemporal information extraction and poor recognition performance in the feature extraction process.
[0126] Example 2
[0127] Please refer to Figure 5 , Figure 5 The figure shows a Transformer-based EEG recognition device provided in this embodiment. In this embodiment, the device includes:
[0128] The EEG signal preprocessing module is used to preprocess the acquired EEG signals to obtain shallow features;
[0129] The spatial feature extraction module is used to input the shallow features into the brain region Transformer module, extract the global spatial features, and obtain the first feature;
[0130] A time feature extraction module, used for inputting the first feature into a time series transformer module, extracting a global time series feature, and obtaining a second feature;
[0131] A deep feature extraction module is used to input the second feature into the temporal transformer module, extract the deep spatiotemporal feature, and obtain the third feature;
[0132] The feature recognition and classification module is used to identify and classify the third feature using a two-layer fully connected neural network and a Softmax layer to obtain a prediction result.
[0133] Optionally, the EEG signal preprocessing module includes:
[0134] A quantization encoding unit, used for quantizing and encoding the acquired EEG signal of the subject and converting it into a continuous digital signal;
[0135] A down-sampling unit, used for down-sampling the digital signal to obtain a digital sampling signal;
[0136] A signal processing unit is used to perform bandpass filtering, independent component analysis and overlapping slicing processing on the digital sampling signal in sequence to obtain a two-dimensional matrix numerical signal;
[0137] The shallow feature acquisition unit is used to perform one-dimensional convolution operations and reshaping operations on the two-dimensional matrix numerical signal to obtain shallow features.
[0138] Optionally, the spatial feature extraction module includes:
[0139] A feature division unit, used to divide shallow features based on the distribution of brain regions and the position distribution of EEG electrodes; the shallow features are signals of several electrodes;
[0140] The intra-region feature learning unit is used to send the divided shallow features to the brain region Transformer module of the corresponding brain region to learn the intra-region spatial features of the brain region;
[0141] An intra-region feature stitching unit is used to stitch the intra-region spatial features of different brain regions to obtain the intra-region spatial features of the entire brain;
[0142] The global feature learning unit is used to send the global spatial features of the entire brain region into the brain region Transformer module, learn the spatial features of the entire brain region, and obtain the global spatial features.
[0143] Optionally, the brain region distribution includes 6 areas, namely, frontal region, left temporal region, right temporal region, central region, parietal region, and occipital region; the position distribution of the EEG electrodes is 7 electrodes in the frontal region, 5 electrodes in the left temporal region, 5 electrodes in the right temporal region, 5 electrodes in the central region, 5 electrodes in the parietal region, and 3 electrodes in the occipital region, a total of 30 electrodes.
[0144] Optionally, the time feature extraction module includes:
[0145] A feature transposition unit, used for transposing the first feature to obtain a transposed feature;
[0146] The global learning unit is used to input the transposed features into the time series Transformer module for global weighted learning, extract the global time series features, and obtain the second features.
[0147] Optionally, also include:
[0148] The dimension expansion unit is used to expand the dimension of the second feature and reshape it into a three-dimensional second feature.
[0149] Optional, deep feature extraction modules include:
[0150] A deep spatial feature extraction unit, used for inputting the second feature into three spatial multi-scale convolutional layers, learning the deep spatial information of the EEG electrode signal from local and overall perspectives, and obtaining deep spatial features of three scales;
[0151] The deep spatial feature concatenation unit is used to spatially concatenate the deep spatial features of three scales in parallel to obtain high-level spatial features;
[0152] The deep temporal feature concatenation unit is used to input high-level spatial features into three temporal convolutional layers of different scales, learn to fuse high-level temporal information, and obtain high-level spatiotemporal features, i.e., the third feature.
[0153] Optionally, also include:
[0154] The feature fusion unit is used to perform 1×1 convolution and 4×4 pooling operations on the third feature to obtain the final fused feature.
[0155] Optionally, also include:
[0156] The result evaluation unit is used to evaluate the predicted result and the acquired true result by using a cross entropy loss function to obtain an evaluation result.
[0157] Based on this, the device uses the brain region Transformer module to capture EEG local and global spatial features from within the brain region and the overall brain region, and at the same time learns the spatial connections between electrodes in different brain regions and on the overall brain region; the time Transformer module is used to fully extract the global temporal dependencies within different time segments, effectively retaining the long-distance temporal signal characteristics, and the spatiotemporal multi-scale convolution module is used to further learn the time and space features to obtain advanced EEG spatiotemporal features, effectively suppressing the category shift problem caused by multi-category sample data, greatly reducing the false detection rate, and can accurately identify EEG signals, thereby solving the problems of incomplete spatiotemporal information extraction and poor recognition performance in the feature extraction process.
[0158] Example 3
[0159] Please refer to Figure 6, this embodiment provides an electronic device, which includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a Transformer-based EEG recognition method at the logical level. Of course, in addition to software implementation methods, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0160] The network interface, processor and memory can be connected to each other through a bus system. The above bus can be divided into an address bus, a data bus, a control bus, etc.
[0161] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor.
[0162] The processor is used to execute the program stored in the above memory, and specifically perform:
[0163] Step 102, preprocessing the acquired EEG signal to obtain shallow features;
[0164] Step 104, inputting the shallow features into the brain region Transformer module, extracting the global spatial features, and obtaining the first features;
[0165] Step 106, input the first feature into the time series Transformer module, extract the global time series feature, and obtain the second feature;
[0166] Step 108, input the second feature into the temporal Transformer module, extract the deep spatiotemporal features, and obtain the third feature;
[0167] Step 110, using a two-layer fully connected neural network and a Softmax layer to identify and classify the third feature to obtain a prediction result.
[0168] The processor may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be completed by an integrated logic circuit of the processor hardware or by instructions in the form of software.
[0169] Based on the same invention, the embodiment of this specification also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1-Figure 4The corresponding embodiment provides a Transformer-based EEG recognition method.
[0170] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-usable program code.
[0171] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.
[0172] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between it and other embodiments.
[0173] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the specific order or sequential order shown in the process depicted in the drawings is not necessarily required to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0174] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A Transformer-based EEG recognition method, characterized in that: include: Preprocess the acquired EEG signals to obtain shallow features; The shallow features are input into the brain region Transformer module to extract the global spatial features and obtain the first feature; The first feature is input into the time series Transformer module, the global time series feature is extracted, and the second feature is obtained; The second feature is input into the temporal transformer module to extract deep spatiotemporal features and obtain the third feature; A two-layer fully connected neural network and a Softmax layer are used to identify and classify the third feature to obtain the prediction result.
2. The method according to claim 1, characterized in that The preprocessing of the acquired EEG signal to obtain shallow features includes: The acquired EEG signals of the subjects are quantized and encoded and converted into continuous digital signals; Down-sampling the digital signal to obtain a digital sampling signal; The digital sampling signal is subjected to bandpass filtering, independent component analysis and overlapping slicing processing in sequence to obtain a two-dimensional matrix numerical signal; Perform one-dimensional convolution and reshaping operations on the two-dimensional matrix numerical signal to obtain shallow features.
3. The method according to claim 1, characterized in that: The inputting of shallow features into the brain region Transformer module to extract global spatial features and obtain the first feature includes: Based on the distribution of brain regions and the position distribution of EEG electrodes, shallow features are divided; the shallow features are signals of several electrodes; The divided shallow features are sent to the brain region Transformer modules of the corresponding brain regions to learn the intra-regional spatial features of the brain regions; The intra-regional spatial features of different brain regions are spliced together to obtain the intra-regional spatial features of the entire brain; The global spatial features of the entire brain region are sent to the brain region Transformer module to learn the spatial features of the entire brain region and obtain the global spatial features.
4. The method according to claim 3, characterized in that The brain region distribution includes 6 areas, namely the frontal region, left temporal region, right temporal region, central region, parietal region, and occipital region; the position distribution of the EEG electrodes is 7 electrodes in the frontal region, 5 electrodes in the left temporal region, 5 electrodes in the right temporal region, 5 electrodes in the central region, 5 electrodes in the parietal region, and 3 electrodes in the occipital region, a total of 30 electrodes.
5. The method according to claim 1, characterized in that Inputting the first feature into the time series Transformer module, extracting the global time series feature, and obtaining the second feature includes: Transposing the first feature to obtain a transposed feature; The transposed features are input into the time series Transformer module for global weighted learning, and the global time series features are extracted to obtain the second features.
6. The method according to claim 5, characterized in that After the transposed features are input into the time series Transformer module for global weighted learning and the global time series features are extracted to obtain the second features, the method further includes: dimensional expansion of the second features to reshape them into three-dimensional second features.
7. The method according to claim 1, characterized in that The step of inputting the second feature into the temporal transformer module, extracting the deep spatiotemporal feature, and obtaining the third feature comprises: Inputting the second feature into three spatial multi-scale convolutional layers, respectively learning the deep spatial information of the EEG electrode signal from the local and overall perspectives, and obtaining deep spatial features of three scales; The deep spatial features of the three scales are spatially paralleled to obtain high-level spatial features; The high-level spatial features are input into three temporal convolutional layers of different scales to learn and fuse the high-level temporal information to obtain high-level spatiotemporal features, i.e., the third feature.
8. The method according to claim 7, characterized in that After inputting the second feature into the temporal Transformer module to extract the deep spatiotemporal features and obtain the third feature, the method further includes: applying 1×1 convolution and 4×4 pooling operations to the third feature to obtain the final fusion feature.
9. The method according to claim 1, characterized in that: After the two-layer fully connected neural network and the Softmax layer are used to identify and classify the third feature to obtain the prediction result, the method also includes: using a cross entropy loss function to evaluate the prediction result and the obtained actual result to obtain an evaluation result.
10. A Transformer-based EEG recognition device, characterized in that: include: The EEG signal preprocessing module is used to preprocess the acquired EEG signals to obtain shallow features; The spatial feature extraction module is used to input the shallow features into the brain region Transformer module, extract the global spatial features, and obtain the first feature; A time feature extraction module, used for inputting the first feature into a time series transformer module, extracting a global time series feature, and obtaining a second feature; A deep feature extraction module is used to input the second feature into the temporal transformer module, extract the deep spatiotemporal feature, and obtain the third feature; The feature recognition and classification module is used to identify and classify the third feature using a two-layer fully connected neural network and a Softmax layer to obtain a prediction result.