Method and device suitable for few-channel motor imagery electroencephalogram signal recognition and storage medium

Abstract

The invention discloses a few-channel motor imagery electroencephalogram signal recognition method and device and a storage medium. Few-channel motor imagery electroencephalogram signals are obtained and filtered through a second-order Butterworth low-pass filter, and continuous wavelet transform is conducted on the filtered signals to generate a time-frequency graph; connecting the time-frequency diagrams of different channels in channel dimensions to form a channel-dependent multi-layer time-frequency diagram, and dividing the channel-dependent multi-layer time-frequency diagram into a training verification set and a test set; the method comprises the following steps of: pre-training an OfficientNet-B0 network, then deleting a full connection layer of the pre-trained network, adding a channel-dependent multi-layer time-frequency graph classification head, and carrying out transfer learning on a training verification set to obtain a final classifier for motor imagery electroencephalogram signal recognition. According to the method, the time-frequency diagram of the motor imagery electroencephalogram signals is generated through continuous wavelet transformation, the channel-dependent multi-layer time-frequency diagram is designed, the time and frequency characteristics of the motor imagery electroencephalogram signals are reserved, and the recognition accuracy of the motor imagery electroencephalogram signals is improved under the limitation of few channels.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) signal processing, and more particularly to the recognition and classification of few-channel EEG signals in motor imagery tasks, with the aim of improving the performance of a brain-computer interface system under few-channel constraints. The present invention provides a method, device and storage medium suitable for recognizing few-channel EEG signals for motor imagery. Background Art

[0002] EEG signals play an important role in clinical diagnosis, neuroscience and brain-computer interface technology, especially in the study of motor imagery tasks, where they can reveal the neural mechanisms of motor control.

[0003] At present, EEG signal classification based on deep learning can be roughly divided into two categories: multi-channel and few-channel. Based on multi-channel EEG signal classification, each channel is usually regarded as a node, and the brain network diagram is further constructed to fuse EEG signal features. When the number of channels is small, for example, less than 3, the brain network diagram is difficult to construct, and the feature extraction ideas of these methods will degenerate into analyzing the time-frequency characteristics of each single-channel EEG signal separately, and then connecting to machine learning for classification. The classification effect will decrease significantly as the number of channels decreases.

[0004] In recent years, the development of portable EEG recording hardware and the improvement of EEG processing technology have promoted the emergence of new EEG applications in daily life. The application of EEG with few channels, such as 1-3 channels, has improved portability and wearability, while reducing the computational load and speeding up the feedback speed, and has been welcomed in these applications. However, at the same time, EEG signals with few channels lead to a reduction in the amount of information, and many methods suitable for multi-channel EEG signal classification are difficult to use in few-channel scenarios. Therefore, it is necessary to study EEG signal recognition methods suitable for few-channel constraints.

[0005] Zhang et al. [1] proposed a fast adaptive sub-band blind source separation method for short-term few-channel EEG recordings to effectively remove artifacts. Zahra Khademi et al. [2] used three hybrid models to study the single-channel classification of EEG signals in brain-computer interfaces based on motor imagery. However, their research focused on analyzing the time-frequency characteristics of a single channel of EEG signals and did not consider methods for fusing information from few-channel EEG signals. Xu et al. [3] proposed a method for fusing three-channel EEG time-frequency maps on a plane for the classification of few-channel motor imagery EEG signals. However, the method is not very scalable and fails to effectively utilize the advantages of deep learning models. Ali Al-Saegh [4] proposed a new EEG dataset expansion method to solve the problem of small data volume in few-channel EEG signals. However, most of these methods focus on the feature analysis of each single-channel EEG signal and ignore the information fusion between channels.

[0006] How to effectively fuse the information of few-channel EEG signals and give full play to the advantages of deep learning models to improve the accuracy of classification detection is an unresolved problem.

[0007] References

[0008] [1] S.Zhang, B.You,

[0009] [2]Z.Khademi, F.Ebrahimi, and HMKordy, "A transfer learning-based cnnand lstm hybrid deep learning model to classify imagery motor eeg signals," Computers in Biology and Medicine, vol.143, p.105288, 2022. [Online].Available:

[0010] https: / / www.sciencedirect.com / science / article / pii / S0010482522000804

[0011] [3] B.Xu, L.Zhang, A.Song, C.Wu, W.Li, D.Zhang, G.Xu, H.Li, and H.Zeng, "Wavelet transform time-frequency image and convolutional network-based motorimagery eeg classification," IEEE Access, vol.7, pp.6084–6093, 2019.

[0012] [4] A. Al-Saegh, SADawwd, and JMAbdul-Jabbar, “Cutcat: An augmentation method for eeg classification,” Neural Networks, vol.141, pp.433–443, 2021. [Online].Available: https: / / www.sciencedirect.com / science / article / pii / S0893608021002288

[0013] [5] M.Moufassih, O.Tarahi, S.Hamou, S.Agounad, and H.Idrissi Azami, “Spectral feature extraction from eeg based motor imagery using commonspatial patterns,” in 2022nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET), 2022, pp.1–6.

[0014] [6] R.Leeb and C.Brunner, "Bci competition 2008{Graz data and set B}," 2008.[Online].Available:https: / / api.semanticscholar.org / CorpusID:16768813 Summary of the invention

[0015] The problem to be solved by the present invention is that in the classification of few-channel motor imagery EEG signals, the traditional method is difficult to fully capture and utilize the time, frequency and spatial characteristics of motor imagery EEG signals under the condition of limited channels, resulting in poor classification performance. At the same time, the existing deep learning network often requires a large amount of data for training, and the scale of the few-channel motor imagery EEG signal data set is usually small, which limits the learning ability and performance of the network.

[0016] The technical solution of the present invention is: a method for identifying EEG signals suitable for few-channel motor imagery based on transfer learning, comprising the following steps:

[0017] Step 1: Evenly time-segment the original few-channel motor imagery EEG signal, remove the time segment where no motor imagery occurs, and obtain the few-channel motor imagery EEG signal; wherein the original few-channel motor imagery EEG signal is collected through the brain-computer interface of the motor imagery subject, and marked with a label of a specific classification of the motor imagery task;

[0018] Step 2: Use a second-order Butterworth low-pass filter to filter the few-channel motor imagery EEG signal, retain the signal in the frequency range of 8-30 Hz, and remove artifacts;

[0019] Step 3: Perform continuous wavelet transform on the filtered signal and output a two-dimensional time-frequency diagram of time and frequency;

[0020] Step 4: feature-connect the time-frequency graphs of different channels in the channel dimension, and output them as a channel-dependent multi-layer EEG time-frequency graph; the feature connection is to vertically superimpose the time-frequency graphs of different channels, and output them as a channel-dependent multi-layer EEG time-frequency graph, and the channel-dependent multi-layer EEG time-frequency graph has three dimensions, representing time, frequency and channel respectively;

[0021] Step 5: dividing the channel-dependent multi-layer EEG time-frequency map into a training validation set and a test set, and dividing the training validation set into a training set and a validation set;

[0022] Step 6: Use the large-scale image dataset ImageNet to pre-train the EfficientNet-B0 network and retain the pre-trained weights of the EfficientNet-B0 network;

[0023] Step 7: Delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer EEG time-frequency map classification head, and generate an EfficientNet-B0 network suitable for EEG signal classification; the channel-dependent multi-layer EEG time-frequency map classification head includes four layers, the first layer is a global average pooling layer, the second layer is a fully connected layer, and uses a ReLU activation function, the third layer is a Dropout layer with a dropout rate set to 0.5, and the fourth layer is a fully connected layer with two neurons and uses softmax as an activation function;

[0024] Step 8: Input the training validation set in step 4 into the EfficientNet-B0 network suitable for EEG signal classification described in step 7 for transfer learning training. After the training is completed, it is used as the final classifier for few-channel motor imagery EEG signal recognition.

[0025] Furthermore, the method further includes step 9, wherein the final classifier obtained through training is evaluated by using a test set. If the evaluation index does not meet the standard, the accuracy of the final classifier is improved by the following methods:

[0026] Add the data of the original few-channel motor imagery EEG signal in step 1;

[0027] Adjust the number of neurons in the second fully connected layer of the EfficientNet-B0 network structure suitable for EEG signal classification described in step 7;

[0028] Or fine-tune the transfer learning training hyperparameters described in step 8.

[0029] Furthermore, in step 3, the continuous wavelet transform uses Morlet wavelet as wavelet basis function, and the scale range of Morlet wavelet is set between 1 and 64, which is used to capture the frequency characteristics of motor imagery EEG signals of 8-30 Hz. After calculation, the continuous wavelet transform outputs a two-dimensional matrix, in which one dimension represents time and the other dimension represents frequency.

[0030] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned final classifier suitable for the few-channel motor imagery EEG signal recognition method, which is used for the recognition of few-channel motor imagery EEG signals.

[0031] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the final classifier suitable for the above-mentioned method for identifying EEG signals of few-channel motor imagery is implemented.

[0032] Compared with the prior art, the method of the present invention generates a time-frequency diagram of the motor imagery EEG signal through continuous wavelet transform, and designs a new channel-dependent multi-layer time-frequency diagram, which retains the time and frequency characteristics of the motor imagery EEG signal, enhances the correlation between channels, and forms a more comprehensive and rich feature representation of the few-channel motor imagery EEG signal. On the other hand, in response to the problem of few channels, the present invention adopts a transfer learning method, in which the EEG signal domain is quite different from the natural image domain used for pre-training of the backbone network, which will affect the transfer learning effect, and the channel-dependent multi-layer time-frequency diagram designed by the present invention reduces the domain difference between the EEG signal domain and the natural image domain, providing a feasibility basis for transfer learning. The present invention first uses the large-scale image dataset ImageNet to pre-train the EfficientNet-B0 network, then designs the classification head of the network according to the EEG signal domain, and transfers the pre-trained model to the few-channel motor imagery EEG data for fine-tuning, thereby overcoming the limitation of the small scale of few-channel motor imagery EEG signal data on the performance of deep learning networks, shortening the model training time, and improving the generalization ability of the model; through the two-level learning framework of "pre-training + fine-tuning", the pre-training knowledge is fully utilized, the model training process of few-channel motor imagery EEG signal classification is accelerated, and the computing resource consumption caused by early trial and error is reduced.

[0033] The present invention improves the recognition accuracy of motor imagery EEG signals under the limitation of few channels. Compared with the 77.22% average classification accuracy achieved by M. Moufassih et al. [5] on the BCI Competition IV dataset 2b [6] of few-channel motor imagery EEG public dataset, the present invention improves the average classification accuracy on the few-channel motor imagery EEG public dataset to 80.21%. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a system architecture diagram of the method of the present invention.

[0035] Figure 2 It is a schematic diagram of constructing the channel-dependent multi-layer EEG time-frequency map described in the present invention.

[0036] Figure 3 It is a schematic diagram of the EfficientNet-B0 network transfer learning strategy suitable for EEG signal classification described in the present invention. DETAILED DESCRIPTION

[0037] Traditional EEG signal feature extraction methods, such as time domain feature extraction, frequency domain feature extraction, and time-frequency feature extraction represented by wavelet transform analysis, are difficult to provide a three-dimensional comprehensive representation of the time, frequency and spatial features of motor imagery EEG signals when EEG channels are limited. The present invention proposes a method for comprehensively capturing and utilizing these features when channels are limited, and vertically superimposes time-frequency graphs of different channels to achieve feature enhancement, so as to achieve accurate classification and recognition of motor imagery EEG signals in a few-channel scenario. Figure 1 As shown, the implementation of the present invention is specifically described below.

[0038] Step 1: Evenly time-segment the original few-channel motor imagery EEG signal to remove the time segment where no motor imagery occurs; wherein the few-channel motor imagery EEG signal is collected via the brain-computer interface of the motor imagery subject and contains a label for the specific classification of the motor imagery task.

[0039] Step 2: Use a second-order Butterworth low-pass filter to filter the few-channel motor imagery EEG signal, retain the signal in the frequency range of 8-30 Hz, and remove artifacts.

[0040] Step 3: Perform continuous wavelet transform on the filtered signal and output it as a time-frequency diagram; the continuous wavelet transform uses Morlet wavelet as the wavelet basis function, and the scale range of Morlet wavelet is set between 1 and 64 to capture the frequency characteristics of motor imagery EEG signals of 8-30Hz. After calculation, the continuous wavelet transform outputs a two-dimensional matrix, one dimension of which represents time and the other dimension represents frequency.

[0041] When processing EEG signals, the prior art obtains time domain features through filtering and other processes, and then uses neural networks for classification. However, this type of method only considers the time domain features, and simple time domain features are difficult to provide a richer feature representation for EEG signals. Moreover, most of the prior art relies on the number of EEG channels to complete the recognition task. As the number of channels decreases, the performance of traditional neural network classification methods decreases, and it is difficult to use in scenarios with few channels, or the effect is not good. In this regard, the present invention not only considers the time domain characteristics of EEG signals, but also obtains two sets of features, time and frequency, through continuous wavelet transform, and further constructs them into channel-dependent multi-layer time-frequency diagrams in subsequent steps, which retains the time and frequency characteristics of motor imagery EEG signals, enhances the correlation between channels, and can better express the time, frequency, and channel characteristics of EEG signals under the restriction of few channels.

[0042] Step 4: Feature-connect the time-frequency graphs of different channels in the channel dimension and output them as channel-dependent multi-layer EEG time-frequency graphs; Figure 2As shown, the process of feature connection is to vertically superimpose the time-frequency diagrams of the different channels and output them as channel-dependent multi-layer EEG time-frequency diagrams. The channel-dependent multi-layer EEG time-frequency diagram has three dimensions, representing time, frequency and channel, respectively, to form a more comprehensive and rich feature representation of the few-channel motor imagery EEG signal.

[0043] Existing few-channel EEG signal classification methods focus on the feature analysis of each single-channel EEG signal, ignoring the information fusion between channels. The present invention obtains the time-frequency representation of EEG signals through continuous wavelet transform, and then vertically superimposes the time-frequency graphs of different channels. The three dimensions of the obtained channel-dependent multi-layer EEG time-frequency graph represent time, frequency and channel respectively, which cleverly and simply forms a more comprehensive and rich feature representation of few-channel motor imagery EEG signals. The present invention provides a three-dimensional image representation of EEG signals through feature enhancement means through feature connection, which reduces the domain difference between the EEG signal domain and the natural image domain, so that the subsequent steps of transfer learning can show better results.

[0044] Step 5: randomly divide the channel-dependent multi-layer EEG time-frequency map into a training validation set and a test set according to a ratio of 9:1, and randomly divide the training validation set into a training set and a validation set according to a ratio of 9:1.

[0045] Step 6: Use the large-scale image dataset ImageNet to pre-train the EfficientNet-B0 network, and retain the pre-trained weights of the EfficientNet-B0 network.

[0046] Step 7: Delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer EEG time-frequency map classification head, and generate an EfficientNet-B0 network suitable for EEG signal classification; the channel-dependent multi-layer EEG time-frequency map classification head includes four layers, the first layer is a global average pooling layer, the second layer is a fully connected layer, and uses a ReLU activation function. In this embodiment, a fully connected layer with 128 neurons is used, the third layer is a Dropout layer with a loss rate set to 0.5, and the fourth layer is a fully connected layer with two neurons and uses softmax as the activation function, such as Figure 3 shown.

[0047] Step 8: The training validation set described in step 4 is input into the EfficientNet-B0 network suitable for EEG signal classification described in step 7 for transfer learning training, and is used as the final classifier after the training is completed; the Adam optimizer is used for training optimization during the transfer learning training; an early stopping strategy is used, and if the loss of the validation set fails to decrease after 20 consecutive epochs, the transfer learning training is terminated.

[0048] Step 9: After the training is completed, the model is evaluated using the test set. If the evaluation index does not meet the standard, the accuracy is improved by adding the data of the original few-channel motor imagery EEG signal in step 1, adjusting the EfficientNet-B0 network structure suitable for EEG signal classification in step 7, or further fine-tuning the transfer learning training hyperparameters in step 8. The evaluation indicators include accuracy, recall and F1 score. Among them, the adjustment of the EfficientNet-B0 network structure suitable for EEG signal classification can adjust the number of neurons in the second fully connected layer. If you want to improve the accuracy, you can try to increase the number based on 128 neurons. If you want to increase the model inference speed, you can reduce the number based on 128 neurons.

[0049] The channel-dependent multi-layer time-frequency diagram designed by the present invention retains the time and frequency characteristics of the motor imagery EEG signal, enhances the correlation between channels, and can better express the time, frequency, and channel characteristics of the EEG signal under the limitation of few channels. In existing studies, transfer learning is rarely used in the field of EEG signal recognition. The reason is that the EEG signal domain is quite different from the natural image domain used for backbone network pre-training. The new channel-dependent multi-layer time-frequency diagram proposed by the present invention reduces the domain difference between the EEG signal domain and the natural image domain, so it can show better effect in transfer learning. After experiments, the present invention surpasses other existing few-channel motor imagery EEG signal detection methods on the few-channel motor imagery EEG public dataset. Compared with the 77.22% average classification accuracy achieved by M. Moufassih et al. [5] on the few-channel motor imagery EEG public dataset BCI Competition IV dataset 2b [6], the average classification accuracy of the present invention on the few-channel motor imagery EEG public dataset is improved to 80.21%.

[0050] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the final classifier of the method for identifying few-channel motor imagery EEG signals based on transfer learning described in the above embodiment, which is used for identifying few-channel motor imagery EEG signals.

[0051] The embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, implements the final classifier suitable for the above-mentioned method for identifying EEG signals of motor imagery with few channels. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

Claims

1. A method for identifying EEG signals of motor imagery with few channels, characterized by: The following steps are involved: Step 1: Evenly time-segment the original few-channel motor imagery EEG signal, remove the time segment where no motor imagery occurs, and obtain the few-channel motor imagery EEG signal; wherein the original few-channel motor imagery EEG signal is collected through the brain-computer interface of the motor imagery subject, and marked with a label of a specific classification of the motor imagery task; Step 2: Use a second-order Butterworth low-pass filter to filter the few-channel motor imagery EEG signal, retain the signal in the frequency range of 8-30 Hz, and remove artifacts; Step 3: Perform continuous wavelet transform on the filtered signal and output a two-dimensional time-frequency diagram of time and frequency; Step 4: feature-connect the time-frequency graphs of different channels in the channel dimension, and output them as a channel-dependent multi-layer EEG time-frequency graph; the feature connection is to vertically superimpose the time-frequency graphs of different channels, and output them as a channel-dependent multi-layer EEG time-frequency graph, and the channel-dependent multi-layer EEG time-frequency graph has three dimensions, representing time, frequency and channel respectively; Step 5: dividing the channel-dependent multi-layer EEG time-frequency map into a training validation set and a test set, and dividing the training validation set into a training set and a validation set; Step 6: Use the large-scale image dataset ImageNet to pre-train the EfficientNet-B0 network and retain the pre-trained weights of the EfficientNet-B0 network; Step 7: Delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer EEG time-frequency map classification head, and generate an EfficientNet-B0 network suitable for EEG signal classification; the channel-dependent multi-layer EEG time-frequency map classification head includes four layers, the first layer is a global average pooling layer, the second layer is a fully connected layer, and uses a ReLU activation function, the third layer is a Dropout layer with a dropout rate set to 0.5, and the fourth layer is a fully connected layer with two neurons and uses softmax as an activation function; Step 8: Input the training validation set in step 4 into the EfficientNet-B0 network suitable for EEG signal classification described in step 7 for transfer learning training. After the training is completed, it is used as the final classifier for few-channel motor imagery EEG signal recognition.

2. The method for identifying EEG signals suitable for motor imagery with few channels according to claim 1, characterized in that The method also includes step 9, in which the final classifier obtained through training is evaluated using the test set. If the evaluation index does not meet the standard, the accuracy of the final classifier is improved by the following methods: Add the data of the original few-channel motor imagery EEG signal in step 1; Adjust the number of neurons in the second fully connected layer of the EfficientNet-B0 network structure suitable for EEG signal classification described in step 7; Or fine-tune the transfer learning training hyperparameters described in step 8.

3. The method for identifying EEG signals suitable for motor imagery with few channels according to claim 2, characterized in that In step 9, the evaluation indicators include precision, recall and F1 score.

4. The method for identifying EEG signals suitable for motor imagery with few channels according to claim 1 or 2, characterized in that In step 3, the continuous wavelet transform uses Morlet wavelet as the wavelet basis function, and the scale range of Morlet wavelet is set between 1 and 64 to capture the frequency characteristics of motor imagery EEG signals of 8-30 Hz. After calculation, the continuous wavelet transform outputs a two-dimensional matrix, in which one dimension represents time and the other dimension represents frequency.

5. The method for recognizing EEG signals suitable for motor imagery with few channels according to claim 1 or 2, The feature is that in step 8, the transfer learning training process uses the Adam optimizer for training optimization; Using the early stopping strategy, if the loss of the validation set fails to decrease after 20 consecutive epochs, the transfer learning training is terminated.

6. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the final classifier suitable for the few-channel motor imagery EEG signal recognition method described in any one of claims 1-5, which is used for the recognition of few-channel motor imagery EEG signals.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, the final classifier suitable for the few-channel motor imagery EEG signal recognition method as described in any one of claims 1-5 is implemented.

Images