A motion imagination electroencephalogram classification method based on hierarchical feature re-labeling network
Patent Information
- Application Number
- CN202311698740.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-12
- Publication Date
- 2026-09-08
- Estimated Expiration
- 2043-12-12
AI Technical Summary
但是，该类方法忽略了时-频-空特征的层次结构，使得分类性能较差
[0005]The beneficial technical effects of the present invention are as follows: The motor imagery EEG classification method based on hierarchical feature recalibration network of the present invention establishes a hierarchical feature recalibration network including a first weighted network for assigning weights to time-frequency unit features and a second weighted network for assigning weights to time window features, so as to achieve hierarchical allocation of weights to each layer of features according to the hierarchical structure of features, fully consider the hierarchical structure of time-frequency-spatial features, better learn the useful time, frequency and spatial prior information of time-frequency-spatial features, and improve the EEG classification performance of motor imagery.
Smart Images
Figure CN117679046B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of motor imagery EEG classification technology, and in particular to a motor imagery EEG classification method based on a hierarchical feature recalibration network. Background Technology
[0002] Numerous studies have demonstrated the close correlation between electroencephalography (EEG) signals and time, frequency, and space. Therefore, time-frequency-spatial (TFS) feature selection methods are increasingly common in EEG classification to learn effective information from TFS features. The extraction process involves segmenting the raw EEG signal into multiple time windows, then decomposing each time window into multiple frequency sub-bands. All time windows and their corresponding frequency sub-bands are combined to form corresponding time-frequency units. Finally, the spatial features of each time-frequency unit are extracted to form a hierarchical structure of TFS features. Traditional TFS feature selection methods concatenate the spatial features of multiple time-frequency units into a single feature vector, treating each feature vector as an independent entity and assigning it a separate weight to learn useful temporal, frequency, and spatial information. However, this type of method neglects the hierarchical structure of TFS features, resulting in poor classification performance. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a motor imagery EEG classification method based on a hierarchical feature recalibration network, so as to improve the classification performance of motor imagery EEG.
[0004] To solve the above-mentioned technical problems, the purpose of this invention is achieved through the following technical solution: A method for classifying motor imagery EEG based on a hierarchical feature recalibration network is provided, comprising the following steps: Time-frequency-spatial feature extraction: The original EEG signal is segmented using time windows and then decomposed according to frequency sub-bands to obtain EEG time-frequency data. Each time window forms a time-frequency unit based on the frequency sub-band. Spatial features are extracted from each time window, each time-frequency unit, and the obtained EEG time-frequency data to obtain time window features, time-frequency unit features, and EEG spatial features, which are then combined to construct a three-layer tree-like distribution of time-frequency-spatial features. A hierarchical feature recalibration network is established: A first weighted network is constructed to assign weights to each time-frequency unit feature to obtain recalibrated time-frequency unit features. A second weighted network is constructed to reassemble the obtained recalibrated time-frequency unit features according to time windows to obtain reassembled time window features. Weights are assigned to each reassembled time window feature to obtain the final calibration features. Classification is performed using a classification module based on the obtained final calibration features to obtain the classification result. The hierarchical feature recalibration network is trained: The hierarchical feature recalibration network is trained.
[0005] The beneficial technical effects of the present invention are as follows: The motor imagery EEG classification method based on hierarchical feature recalibration network of the present invention establishes a hierarchical feature recalibration network including a first weighted network for assigning weights to time-frequency unit features and a second weighted network for assigning weights to time window features, so as to achieve hierarchical allocation of weights to each layer of features according to the hierarchical structure of features, fully consider the hierarchical structure of time-frequency-spatial features, better learn the useful time, frequency and spatial prior information of time-frequency-spatial features, and improve the EEG classification performance of motor imagery. Attached Figure Description
[0006] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0007] Figure 1 This is a schematic diagram of the data processing flow of the motor imagery EEG classification method based on hierarchical feature recalibration network provided in an embodiment of the present invention;
[0008] Figure 2 This is a schematic diagram of the hierarchical structure of time-frequency-spatial features in the time-frequency-spatial feature extraction of the motor imagery EEG classification method based on hierarchical feature recalibration network provided in the embodiments of the present invention. Detailed Implementation
[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0010] Please see Figure 1 , Figure 1 This is a schematic diagram of the data processing flow of the motor imagery EEG classification method based on hierarchical feature recalibration networks provided in an embodiment of the present invention. The motor imagery EEG classification method based on hierarchical feature recalibration networks includes the following steps:
[0011] Step S11, Time-Frequency-Spatial Feature Extraction: The original EEG signal is segmented using time windows and then decomposed according to frequency sub-bands to obtain EEG time-frequency data. Each time window forms a time-frequency unit based on the frequency sub-band. Spatial features are extracted from each time window, each time-frequency unit, and the obtained EEG time-frequency data to obtain time window features, time-frequency unit features, and EEG spatial features, which are then combined to construct a three-layer tree-like time-frequency-spatial feature structure. Each time window corresponds to multiple frequency sub-bands, and the time window and the corresponding frequency sub-band can form multiple time-frequency units. The number of time-frequency units is the product of the number of frequency sub-bands and the number of time windows. The time window features correspond one-to-one with the time window, and the time-frequency unit features correspond one-to-one with the time-frequency unit.
[0012] Step S12: Establish a hierarchical feature recalibration network: Construct a first weighted network to assign weights to each time-frequency unit feature to obtain the time-frequency unit features during recalibration; construct a second weighted network to reassemble the obtained time-frequency unit features according to a time window to obtain the reassembled time window features; assign weights to each reassembled time window feature to obtain the final calibration features; and use a classification module to classify based on the obtained final calibration features to obtain the classification results.
[0013] Step S13: Train the hierarchical feature relabeling network: Train the hierarchical feature relabeling network.
[0014] The motor imagery EEG classification method based on hierarchical feature recalibration network establishes a hierarchical feature recalibration network, which includes a first weighted network that assigns weights to time-frequency unit features and a second weighted network that assigns weights to time window features. This allows for hierarchical weight allocation of features at each layer according to their hierarchical structure, fully considering the hierarchical structure of time-frequency-spatial features, and better learning the useful time, frequency, and spatial prior information of time-frequency-spatial features, thereby improving the EEG classification performance of motor imagery.
[0015] Specifically, step S11 includes:
[0016] The raw EEG signal was segmented using time windows to obtain multiple time-window EEG signals. Each time window has a window size of 2 seconds, and the start time of each time window increases sequentially, with a time difference of 0.5 seconds between the start times of adjacent time windows. The first time window can be 0–2 seconds, the second time window can be 0.5–2.5 seconds, and so on. 0 seconds represents the start time of the motor imagery task.
[0017] The EEG signals for each time window are decomposed according to frequency sub-bands to obtain EEG time-frequency data. Each time window forms a time-frequency unit based on the frequency sub-bands. There are multiple frequency sub-bands, with a frequency interval of 4 Hz between each sub-band. The first frequency of each sub-band increases sequentially, and the frequency difference between the first frequencies of two adjacent sub-bands is 2 Hz. The first frequency sub-band can be 4–8 Hz, the second can be 6–10 Hz, and so on, with the last sub-band being 36–40 Hz.
[0018] Spatial features were extracted from each time window, each time-frequency unit, and the acquired EEG time-frequency data using the Common Spatial Pattern (CSP) method. This yielded time window features, time-frequency unit features, and EEG spatial features. The obtained time window features, time-frequency unit features, and EEG spatial features were combined to construct a three-layer tree-like time-frequency-space feature structure, recording the hierarchical structure of the time-frequency-space features. The logarithm of the spatial filter in the Common Spatial Pattern method was set to 1. Figure 2 This demonstrates the hierarchical structure of time-frequency-spatial features during the extraction process, such as... Figure 2 As shown, the entire time-frequency-space feature extraction process is implemented layer by layer. Using the original EEG signal as the root node and time windows and frequency sub-bands as intermediate nodes, the time window is the first layer of the tree, and the time window features are the features of the first layer. Their weights are denoted as... The superscript represents the first level, and the subscript represents the time window number. The frequency subband is considered the second level of the tree, so the time-frequency unit features are the features of the second level, and their weights are denoted as... The superscript represents the second layer, and the subscript represents the combination of the time window number and the frequency sub-band number; the EEG spatial features are used as leaf nodes, that is, the specific individual spatial features of the EEG time-frequency data in each time-frequency unit are used as leaf nodes, and their weights are denoted as... The superscript represents the third layer, and the subscript represents the combination of the time window number, the frequency sub-band number, and the spatial feature number. m represents the logarithm of the spatial filter in the co-space mode method. m=1, that is, the specific individual spatial features of each time-frequency unit feature are taken as the third layer of the tree. Thus, the EEG spatial features are the third layer of the tree, constructing a three-layer tree-like feature structure. Therefore, the time-frequency-space features have a three-layer tree-like feature structure.
[0019] Specifically, step S12 includes:
[0020] Construct the first weighted network: Perform compression, excitation, and feature rescaling operations on the features of each time-frequency unit in sequence to obtain the recalibrated time-frequency unit features; wherein, the compression ratio of the excitation operation in the first weighted network can be 17.
[0021] Construct a second weighted network: Reorganize the obtained recalibrated time-frequency unit features according to the time window to obtain reorganized time window features. Perform squeezing, excitation and feature rescaling operations on each reorganized time window feature in sequence to obtain the final calibration features; wherein, the compression ratio of feature rescaling in the second weighted network can be 2.
[0022] Feature Classification: Based on the obtained final labeled features, a classification module is used for classification to obtain the classification result. The classification module includes a flattened layer, a concatenated layer, a dropout layer, and a fully connected layer arranged sequentially. Each fully connected layer consists of two neurons, with the sigmoid activation function. The classification module selects binary cross-entropy as the loss function and uses binary classification accuracy as the evaluation metric.
[0023] Specifically, the steps of constructing the first weighted network include:
[0024] Each time-frequency unit feature is treated as a feature map;
[0025] The feature map is squeezed to construct the channel descriptor of the feature map;
[0026] Among them, global average pooling can be used to construct the channel descriptor of the feature map; then the channel descriptor of the feature map can be calculated using formula (1):
[0027]
[0028] In the formula, z c Let x be the c-th element of the channel descriptor z of the feature map, H1 and W1 represent the dimensions of the feature map, C1 represent the number of channels in all feature maps, H1 = 2m, where m represents the logarithm of the spatial filters in the co-space mode method, m = 1, then H1 = 2, W1 = 1, C1 = T × F, where T represents the number of time windows, F represents the number of frequency sub-bands corresponding to a single time window, and x c (i,j) represents the characteristics of the time-frequency unit. R represents the range of real numbers.
[0029] Alternatively, global max pooling can be used to construct the channel descriptors of the feature map; then the channel descriptors of the feature map can be calculated using formula (2):
[0030] z c =max(x c (i,j)),c=1,2,...,C1,i=1,2,...,H1,j=1,2,...,W1 (2)
[0031] In the formula, z cLet x represent the c-th element of the channel descriptor z, H1 and W1 represent the dimensions of the feature maps, C1 represent the number of channels in all feature maps, H1 = 2m, where m represents the logarithm of the spatial filters in the co-space mode method, m = 1, then H1 = 2, W1 = 1, C1 = T × F, where T represents the number of time windows, F represents the number of frequency sub-bands corresponding to a single time window, and x c (i,j) represents the characteristics of the time-frequency unit. R represents the range of real numbers.
[0032] The feature map is activated, and the channel weights of the feature map are calculated.
[0033] Specifically, the step of performing an activation operation on the feature map and calculating the channel weights of the feature map is as follows:
[0034] Two fully connected layers are used to form a bottleneck network to learn the interdependencies between channels in the feature maps. The channel weights of the feature maps are calculated using formula (3):
[0035] s=σ(W2δ(W1z)) (3)
[0036] In the formula, s represents the channel weights of the feature map, δ represents the Exponential Linear Unit (ELU) activation function, σ represents the sigmoid activation function, z represents the channel descriptor of the feature map, W1 represents the dimensionality reduction layer, and W2 represents the dimensionality increase layer. R represents the range of real numbers, C1 represents the number of channels in all feature maps, r1 represents the first compression ratio, and the value of r1 can be set according to requirements. In this embodiment, r1 = 17.
[0037] The feature maps are scaled according to their channel weights to assign weights to the features of each time-frequency unit, resulting in updated feature maps. These updated feature maps are then used as the recalibrated features of the time-frequency units.
[0038] Specifically, the updated feature map can be calculated using formula (4):
[0039] x′ c =F scale1 (x c ,s c ),c=1,2,...,C1 (4)
[0040] In the formula, x′ c This represents the updated feature map, i.e., the recalibrated time-frequency cell features, F. scale1 (·,·) indicates that the multiplication operation is performed channel by channel, s c x represents the channel weight of the c-th channel of the feature map. c This represents the feature map, i.e., the initial time-frequency unit features, and c represents the channel number.
[0041] Specifically, the steps for constructing the second weighted network include:
[0042] The recalibrated time-frequency unit features obtained and output by the first weighted network are recombined according to a time window to obtain recombined time window features.
[0043] The recombination time window features are squeezed to construct a channel descriptor for the recombination time window features;
[0044] Among them, global average pooling can be used to construct the channel descriptor of the reconstructed time window feature; then the channel descriptor of the reconstructed time window feature can be calculated using formula (5):
[0045]
[0046] In the formula, z″ c The c-th element of the channel descriptor z″ represents the recombined time window feature. H2 and W2 represent the dimensions of the recombined time window feature, C2 represents the number of channels for all recombined time window features, H2 = 2m × F, where m represents the logarithm of the spatial filters in the co-space mode method. If m = 1, then H2 = 2F, W2 = 1, C2 = T, where T represents the number of time windows, and F represents the number of frequency subbands corresponding to a single time window. x″ c (i,j) represents the recombination time window characteristics. R represents the range of real numbers.
[0047] Alternatively, global max pooling can be used to construct the channel descriptors of the feature map; then the channel descriptors of the feature map can be calculated using formula (6):
[0048] z″ c =max(x″) c (i,j)),c=1,2,...,C2,i=1,2,...,H2,j=1,2,...,W2 (6)
[0049] In the formula, z″ c The c-th element of the channel descriptor z″ represents the recombined time window feature. H2 and W2 represent the dimensions of the recombined time window feature, C2 represents the number of channels for all recombined time window features, H2 = 2m × F, where m represents the logarithm of the spatial filters in the co-space mode method. If m = 1, then H2 = 2F, W2 = 1, C2 = T, where T represents the number of time windows, and F represents the number of frequency subbands corresponding to a single time window. x″ c (i,j) represents the recombination time window characteristics. R represents the range of real numbers.
[0050] The recombination time window features are activated, and the channel weights of the recombination time window features are calculated.
[0051] Specifically, the step of activating the recombination time window feature and calculating the channel weights of the recombination time window feature is as follows:
[0052] Two fully connected layers are used to form a bottleneck network to learn the channel interdependencies between the reconstructed time window features. The channel weights of the reconstructed time window features are calculated using formula (7).
[0053] s″=σ(W4δ(W3z″)) (7)
[0054] In the formula, s″ represents the channel weights of the reconstructed time window features, δ represents the Exponential Linear Unit (ELU) activation function, σ represents the sigmoid activation function, z″ represents the channel descriptor of the reconstructed time window features, W3 represents the dimensionality reduction layer, and W4 represents the dimensionality increase layer. R represents the real number range, C2 represents the number of channels for all recombination time window features, r2 represents the second compression ratio, and the value of r2 can be set according to requirements. In this embodiment, r2 = 2.
[0055] The features of each recombination time window are scaled according to the channel weights of the recombination time window features, and weights are assigned to each recombination time window feature to obtain the final calibration features.
[0056] Specifically, the final calibration features can be calculated using formula (8):
[0057]
[0058] In the formula, F represents the final calibration feature. scale2 (·,·) indicates that the multiplication operation is performed channel by channel, s″ c x″ represents the channel weight of the c-th channel representing the recombination time window feature. c This indicates the characteristics of the recombination time window, and c represents the channel number.
[0059] Specifically, the feature classification steps are as follows:
[0060] After the final calibration features are obtained by tiling, a cascaded layer is used to concatenate all the tiled final calibration features into a feature vector. After the feature vector is processed by the dropout layer, the processed feature vector is input into the fully connected layer to obtain the classification result.
[0061] Specifically, step S13 is as follows:
[0062] The hierarchical feature recalibration network was trained using the Adam optimizer with a learning rate of 0.0001. The dropout probability was set to 0.5. An early stopping mechanism was employed: training was stopped if the model loss on the validation set did not decrease for twenty consecutive epochs. The maximum number of training epochs was 1000, and the batch size was 10. The training process was implemented on an NVIDIA RTX 3090 with 24GB of memory using the TensorFlow 2.10 deep learning architecture.
[0063] Preferably, for each subject, hierarchical five-fold cross-validation is used to further divide the original training set into a training subset and a validation set. The hierarchical feature recalibration network model is trained five times, resulting in five models used for classification and prediction on the test set. The average of the five classification results is used as the final classification metric for the subject.
[0064] In summary, the motor imagery EEG classification method based on hierarchical feature recalibration network of the present invention establishes a hierarchical feature recalibration network, which includes a first weighted network that assigns weights to time-frequency unit features and a second weighted network that assigns weights to time window features. This allows for hierarchical weight allocation of features at each layer according to their hierarchical structure, fully considering the hierarchical structure of time-frequency-spatial features, and better learning of useful time, frequency, and spatial prior information of time-frequency-spatial features, thereby improving the EEG classification performance of motor imagery.
[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for classifying motor imagery EEG based on a hierarchical feature recalibration network, characterized in that, Includes the following steps: Time-frequency-spatial feature extraction: The original EEG signal is segmented by time windows and then decomposed according to frequency sub-bands to obtain EEG time-frequency data. Each time window forms a time-frequency unit according to the frequency sub-band. Spatial features are extracted from each time window, each time-frequency unit, and the obtained EEG time-frequency data to obtain time window features, time-frequency unit features, and EEG spatial features, which are then combined to construct a three-layer tree-like time-frequency-spatial feature. Establish a hierarchical feature recalibration network: Construct a first weighted network to assign weights to each time-frequency unit feature to obtain the time-frequency unit features during recalibration; construct a second weighted network to reorganize the obtained time-frequency unit features according to time windows to obtain reorganized time window features; assign weights to each reorganized time window feature to obtain the final calibration features; and use a classification module to classify based on the obtained final calibration features to obtain the classification results. Training the hierarchical feature relabeling network: Train the hierarchical feature relabeling network.
2. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 1, characterized in that, The steps for extracting time-frequency-space features include: The raw EEG signal was segmented using time windows to obtain EEG signals of multiple time windows; The EEG signals of each time window are decomposed according to frequency sub-bands to obtain EEG time-frequency data, and each time window forms a time-frequency unit according to the frequency sub-bands. Spatial features were extracted from each time window, each time-frequency unit, and the obtained EEG time-frequency data using the co-spatial pattern method. Time window features, time-frequency unit features, and EEG spatial features were obtained. The obtained time window features, time-frequency unit features, and EEG spatial features were combined to construct a three-layer tree-like time-frequency-space feature structure, and the hierarchical structure of the time-frequency-space feature was recorded.
3. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 1, characterized in that, The steps for establishing the hierarchical feature recalibration network include: Construct the first weighted network: Perform squeezing, excitation and feature rescaling operations on the features of each time-frequency unit in sequence to obtain the recalibrated time-frequency unit features; Construct a second weighted network: Reorganize the obtained recalibrated time-frequency unit features according to the time window to obtain reorganized time window features. Perform squeezing, excitation and feature rescaling operations on each reorganized time window feature in sequence to obtain the final calibration features. Feature classification: Based on the obtained final calibrated features, a classification module is used to classify the features and obtain the classification results; wherein, the classification module includes a tiling layer, a cascaded layer, a dropout layer and a fully connected layer arranged in sequence.
4. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 3, characterized in that, The steps for constructing the first weighted network include: Each time-frequency unit feature is treated as a feature map; The feature map is squeezed to construct the channel descriptor of the feature map; The feature map is activated, and the channel weights of the feature map are calculated. The feature maps are scaled according to their channel weights to assign weights to the features of each time-frequency unit, resulting in updated feature maps. These updated feature maps are then used as the recalibrated features of the time-frequency units.
5. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 4, characterized in that, The specific steps for performing a squeezing operation on the feature map to construct the channel descriptor of the feature map are as follows: constructing the channel descriptor of the feature map using global average pooling or global max pooling.
6. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 4, characterized in that, The specific steps for performing the activation operation on the feature map and calculating the channel weights of the feature map are as follows: Two fully connected layers are used to form a bottleneck network to learn the channel dependencies between feature maps. The channel weights of the feature maps are calculated using the following formula: s=σ(W2δ(W1z)); In the formula, s represents the channel weights of the feature map, δ represents the exponential linear unit activation function, σ represents the sigmoid activation function, z represents the channel descriptor of the feature map, W1 represents the dimensionality reduction layer, and W2 represents the dimensionality increase layer. R represents the real number range, C1 represents the number of channels in all feature maps, and r1 represents the first compression ratio.
7. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 3, characterized in that, The steps for constructing the second weighted network include: The recalibrated time-frequency unit features obtained from the output of the first weighted network are reorganized according to a time window to obtain reorganized time window features. The recombination time window features are squeezed to construct a channel descriptor for the recombination time window features; The recombination time window features are activated, and the channel weights of the recombination time window features are calculated. The features of each recombination time window are scaled according to the channel weights of the recombination time window features, and weights are assigned to each recombination time window feature to obtain the final calibration features.
8. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 7, characterized in that, The specific steps for activating the recombination time window features and calculating the channel weights of the recombination time window features are as follows: Two fully connected layers are used to form a bottleneck network to learn the channel interdependencies between features of the reconstructed time window. The channel weights of the features of the reconstructed time window are calculated using the following formula: s″=σ(W4δ(W3z″)); In the formula, s″ represents the channel weights of the reconstructed time window features, δ represents the exponential linear unit activation function, σ represents the sigmoid activation function, z″ represents the channel descriptor of the reconstructed time window features, W3 represents the dimensionality reduction layer, and W4 represents the dimensionality increase layer. R represents the real number range, C2 represents the number of channels for all recombination time window features, and r2 represents the second compression ratio.
9. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 3, characterized in that, The specific steps of feature classification are as follows: The final labeled features are flattened and concatenated into a feature vector. The feature vector is then processed using a dropout layer. The processed feature vector is then input into a fully connected layer to obtain the classification result.
10. The motor imagery EEG classification method based on hierarchical feature recalibration network according to claim 1, characterized in that, The specific steps for training the hierarchical feature recalibration network are as follows: The hierarchical feature recalibration network was trained using the Adam optimizer with a learning rate of 0.0001.