Method, device and storage medium suitable for few-channel motor imagery electroencephalogram signal recognition

Abstract

A method and device for recognizing motor imagery electroencephalogram signals with few channels and a storage medium are provided.The motor imagery electroencephalogram signals are acquired, filtered with a second-order Butterworth low-pass filter, and subjected to continuous wavelet transform to generate time-frequency graphs.The time-frequency graphs of different channels are connected in the channel dimension to form channel-dependent multi-layer time-frequency graphs, which are divided into a training verification set and a test set.The EfficientNet-B0 network is pre-trained, the fully connected layer of the pre-trained network is deleted, a channel-dependent multi-layer time-frequency graph classification head is added, and transfer learning is performed on the training verification set to obtain a final classifier for recognizing motor imagery electroencephalogram signals.The time-frequency graphs of motor imagery electroencephalogram signals are generated through continuous wavelet transform, and a channel-dependent multi-layer time-frequency graph is designed, thereby retaining the time and frequency characteristics of motor imagery electroencephalogram signals, and improving the recognition accuracy of motor imagery electroencephalogram signals under the limitation of few channels.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram signal processing, and relates to the recognition and classification of few-channel electroencephalogram signals in motor imagery tasks, aiming to improve the performance of brain-computer interface systems under the restriction of few channels, and is a kind of method, device and storage medium suitable for few-channel motor imagery electroencephalogram signal recognition. BACKGROUND

[0002] Electroencephalogram signals play an important role in clinical diagnosis, neuroscience and brain-computer interface technology, especially in the study of motor imagery tasks, which can reveal the neural mechanisms of motor control.

[0003] At present, the classification of electroencephalogram signals based on deep learning can be roughly divided into two categories: multi-channel and few-channel. The classification of electroencephalogram signals based on multi-channel usually adopts the method of regarding each channel as a node and further constructing a brain network graph to fuse the features of electroencephalogram signals. However, when the number of channels is small, for example, less than 3, it is difficult to construct a brain network graph, and the feature extraction idea of these methods will degenerate into analyzing the time-frequency characteristics of each single-channel electroencephalogram signal separately, and then using machine learning to classify, and the classification effect will decrease significantly with the decrease of the number of channels.

[0004] In recent years, the development of portable electroencephalogram recording hardware and the improvement of electroencephalogram processing technology have promoted the emergence of new electroencephalogram applications in daily life, and the application of few channels, such as 1-3 channels of electroencephalogram, has improved the portability and wearability, while reducing the computational load and speeding up the feedback speed, which is welcomed in these applications. At the same time, the few channels of electroencephalogram signals lead to a reduction in information quantity, and many methods suitable for multi-channel electroencephalogram signal classification are difficult to use in few-channel scenarios, so it is necessary to study the electroencephalogram signal recognition method suitable for few-channel restriction.

[0005] Zhang[1] et al. proposed a fast adaptive sub-band blind source separation method for short-term few-channel electroencephalogram recording to effectively remove artifacts, and Zahra Khademi[2] et al. studied the single-channel classification of electroencephalogram signals in brain-computer interface based on motor imagery using three mixing models, but their research focused on analyzing the time-frequency characteristics of single-channel electroencephalogram signals, and did not consider the method of fusing the information of few-channel electroencephalogram signals. Xu[3] et al. proposed a method of fusing three channels of electroencephalogram time-frequency graphs in a plane for few-channel motor imagery EEG signal classification, but the method has poor scalability and fails to effectively utilize the advantages of deep learning models. Ali Al-Saegh[4] proposed a new method of expanding electroencephalogram data set to solve the problem of insufficient data quantity in few-channel electroencephalogram signals. However, most of these methods focus on feature analysis of each single-channel electroencephalogram signal, ignoring the information fusion between channels.

[0006] How to effectively fuse the information of few-channel EEG signals and give full play to the advantages of deep learning models to improve the accuracy of classification detection is an unresolved problem.

[0007] References

[0008] [1] S.Zhang, B.You,

[0009] [2]Z.Khademi, F.Ebrahimi, and HMKordy, "A transfer learning-based cnnand lstm hybrid deep learning model to classify imagery motor eeg signals," Computers in Biology and Medicine, vol.143, p.105288, 2022. [Online].Available:

[0010] https: / / www.sciencedirect.com / science / article / pii / S0010482522000804

[0011] [3] B.Xu, L.Zhang, A.Song, C.Wu, W.Li, D.Zhang, G.Xu, H.Li, and H.Zeng, "Wavelet transform time-frequency image and convolutional network-based motorimagery eeg classification," IEEE Access, vol.7, pp.6084–6093, 2019.

[0012] [4] A. Al-Saegh, S. A. Dawwd, and J. M. Abdul-Jabbar, “Cutcat: An augmentation method for eeg classification,” Neural Networks, vol. 141, pp. 433-443, 2021. [Online]. Available: https: / / www.sciencedirect.com / science / article / pii / S0893608021002288

[0013] [5] M. Moufassih, O. Tarahi, S. Hamou, S. Agounad, and H. Idrissi Azami, “Spectral feature extraction from eeg based motor imagery using common spatial patterns,” in 2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET), 2022, pp. 1-6.

[0014] [6] R. Leeb and C. Brunner, “Bci competition 2008 {Graz data and set B},” 2008. [Online]. Available: https: / / api.semanticscholar.org / CorpusID:16768813 SUMMARY

[0015] The problem to be solved by the present application is that in the classification of few-channel motor imagery electroencephalogram signals, the traditional method is difficult to fully capture and utilize the time, frequency and spatial characteristics of motor imagery electroencephalogram signals under the condition of limited channels, resulting in low classification performance. At the same time, the existing deep learning network often needs a large amount of data for training, while the size of the few-channel motor imagery electroencephalogram signal dataset is usually small, which limits the learning ability and performance of the network.

[0016] The technical scheme of the present application is: a few-channel motor imagery electroencephalogram signal recognition method based on transfer learning, comprising the following steps:

[0017] Step 1: uniformly time segment the original few-channel motor imagery electroencephalogram signal, remove the time segments where no motor imagery occurs, and obtain the few-channel motor imagery electroencephalogram signal; wherein the original few-channel motor imagery electroencephalogram signal is collected by a brain-computer interface of a motor imagery subject, and labeled with a label of a specific classification of a motor imagery task;

[0018] Step 2: filter the few-channel motor imagery electroencephalogram signal using a second-order Butterworth low-pass filter to retain signals in the frequency range of 8-30 Hz and remove artifacts;

[0019] Step 3: perform continuous wavelet transform on the filtered signal to output a two-dimensional time-frequency graph of time and frequency;

[0020] Step 4: feature connect the time-frequency graphs of different channels in the channel dimension to output a channel-dependent multi-layer electroencephalogram time-frequency graph; the feature connection is to vertically stack the time-frequency graphs of different channels to output a channel-dependent multi-layer electroencephalogram time-frequency graph, which has three dimensions representing time, frequency, and channel;

[0021] Step 5: divide the channel-dependent multi-layer electroencephalogram time-frequency graph into a training and validation set and a test set, and divide the training and validation set into a training set and a validation set;

[0022] Step 6: pre-train the EfficientNet-B0 network using a large-scale image dataset ImageNet and retain the pre-training weights of the EfficientNet-B0 network;

[0023] Step 7: delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer electroencephalogram time-frequency graph classification head, and generate an EfficientNet-B0 network suitable for electroencephalogram signal classification; the channel-dependent multi-layer electroencephalogram time-frequency graph classification head includes four layers, the first layer is a global average pooling layer, the second layer is a fully connected layer with a ReLU activation function, the third layer is a Dropout layer with a loss rate of 0.5, and the fourth layer is a fully connected layer with two neurons and a softmax activation function;

[0024] Step 8: input the training and validation set in step 4 into the EfficientNet-B0 network suitable for electroencephalogram signal classification in step 7 for transfer learning training, and after training, use the final classifier for few-channel motor imagery electroencephalogram signal recognition.

[0025] Further, step 9 is also included, which evaluates the final classifier obtained by training using the test set to verify the evaluation index, and if the evaluation index does not meet the standard, the accuracy of the final classifier is improved by the following method:

[0026] Increasing the data of the original few-channel motor imagery electroencephalogram signal in step 1;

[0027] Adjusting the number of neurons of the second full connection layer in the EfficientNet-B0 network structure suitable for electroencephalogram signal classification in step 7;

[0028] Or fine-tuning the transfer learning training hyperparameters in step 8.

[0029] Further, in step 3, the continuous wavelet transform adopts Morlet wavelet as the wavelet base function, and the scale range of the Morlet wavelet is set between 1 and 64, which is used to capture the frequency characteristics of the motor imagery electroencephalogram signal of 8-30Hz, and the continuous wavelet transform outputs a two-dimensional matrix after calculation, wherein one dimension represents time and the other dimension represents frequency.

[0030] The application further provides an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor, and realizes the final classifier of the above-mentioned few-channel motor imagery electroencephalogram signal recognition method for the recognition of the few-channel motor imagery electroencephalogram signal.

[0031] The application further provides a computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is executed to realize the final classifier of the above-mentioned few-channel motor imagery electroencephalogram signal recognition method.

[0032] Compared with the prior art, the method of the application generates a time-frequency graph of motor imagery electroencephalogram signals by continuous wavelet transform, and designs a new channel-dependent multi-layer time-frequency graph, which retains the time and frequency characteristics of motor imagery electroencephalogram signals, enhances the correlation between channels, and forms a more comprehensive and rich feature representation of the few-channel motor imagery electroencephalogram signals. On the other hand, in view of the problem of few channels, the application adopts a transfer learning method, in which the domain of electroencephalogram signals and the natural image domain used for pre-training of the backbone network have large differences, which will affect the transfer learning effect, and the channel-dependent multi-layer time-frequency graph designed in the application reduces the domain difference between the electroencephalogram signal domain and the natural image domain, providing a feasibility basis for transfer learning. The application first pre-trains the EfficientNet-B0 network using a large-scale image dataset ImageNet, and then designs a network classification head according to the electroencephalogram signal domain, migrates the pre-trained model to the few-channel motor imagery electroencephalogram data for fine-tuning, overcomes the limitation of small-scale few-channel motor imagery electroencephalogram signal data on the performance of deep learning network, shortens the model training time, and improves the generalization ability of the model; through the two-level learning framework of "pre-training + fine-tuning", the pre-trained knowledge is fully utilized, the model training process of few-channel motor imagery electroencephalogram signal classification is accelerated, and the calculation resource consumption caused by early trial and error is reduced.

[0033] Under the limitation of few channels, the application improves the recognition accuracy of motor imagery electroencephalogram signals. Compared with the average classification accuracy of 77.22% achieved by M. Moufassih [5] et al. on the few-channel motor imagery electroencephalogram public dataset BCI Competition IV dataset 2b [6], the average classification accuracy on the few-channel motor imagery electroencephalogram public dataset is improved to 80.21%. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a system architecture diagram of the method of the application.

[0035] Figure 2 is a construction schematic diagram of the channel-dependent multi-layer electroencephalogram time-frequency graph of the application.

[0036] Figure 3 is a transfer learning strategy schematic diagram of the EfficientNet-B0 network suitable for electroencephalogram signal classification of the application. DETAILED DESCRIPTION

[0037] The traditional electroencephalogram signal feature extraction method, such as time domain feature extraction, frequency domain feature extraction, and wavelet transform analysis as represented by time-frequency feature extraction, is difficult to provide a three-dimensional comprehensive representation of the time, frequency and spatial features of the motor imagery electroencephalogram signal under the condition of limited electroencephalogram channels, and the present application proposes a method for comprehensively capturing and utilizing these features under the condition of limited channels, which vertically stacks the time-frequency diagrams of different channels to realize feature enhancement, so as to realize accurate classification and recognition of the motor imagery electroencephalogram signal under the condition of few channels. Figure 1 As shown in the following, the implementation of the present application is specifically described.

[0038] Step 1: uniformly time-dividing the original few-channel motor imagery electroencephalogram signal, removing the time segments in which no motor imagery occurs; wherein the few-channel motor imagery electroencephalogram signal is collected by the brain-computer interface of a motor imagery subject, and contains labels of specific classification of motor imagery tasks.

[0039] Step 2: filtering the few-channel motor imagery electroencephalogram signal using a second-order Butterworth low-pass filter, retaining signals in the frequency range of 8-30 Hz and removing artifacts.

[0040] Step 3: performing continuous wavelet transform on the filtered signal, and outputting a time-frequency diagram; the continuous wavelet transform uses Morlet wavelet as the wavelet basis function, and the scale range of the Morlet wavelet is set between 1 and 64, which is used to capture the frequency characteristics of the motor imagery electroencephalogram signal in the range of 8-30 Hz. The continuous wavelet transform outputs a two-dimensional matrix after calculation, wherein one dimension represents time and the other dimension represents frequency.

[0041] In the prior art, when processing electroencephalogram signals, time domain features are obtained through filtering and other processing, and then a neural network is used for classification, but this method only considers time domain features, and pure time domain features are difficult to provide more rich feature representation of electroencephalogram signals. Moreover, most of the prior art relies on the number of electroencephalogram channels to complete the recognition task, and as the number of channels decreases, the performance of the traditional neural network classification method decreases, and it is difficult to use in the scene under the limitation of few channels, or the effect is not good. Therefore, the present application not only considers the time domain features of the electroencephalogram signal, but also obtains two groups of time and frequency features through continuous wavelet transform, and further constructs a channel-dependent multi-layer time-frequency diagram in the subsequent steps, retains the time and frequency features of the motor imagery electroencephalogram signal, enhances the correlation between channels, and better represents the time, frequency and channel features of the electroencephalogram signal under the limitation of few channels.

[0042] Step 4: connecting the time-frequency diagrams of different channels in the channel dimension, and outputting a channel-dependent multi-layer electroencephalogram time-frequency diagram; as Figure 2As shown, the feature connection process is to stack the time-frequency diagrams of different channels longitudinally, and the output is a channel-dependent multi-layer electroencephalogram time-frequency diagram. The channel-dependent multi-layer electroencephalogram time-frequency diagram has three dimensions, representing time, frequency and channel, respectively, to form a more comprehensive and rich feature representation of the few-channel motor imagery electroencephalogram signal.

[0043] The existing few-channel electroencephalogram signal classification method focuses on the feature analysis of each single-channel electroencephalogram signal, ignoring the information fusion between channels. The present application obtains the time-frequency representation of the electroencephalogram signal through continuous wavelet transform, and then stacks the time-frequency diagrams of different channels longitudinally. The three dimensions of the obtained channel-dependent multi-layer electroencephalogram time-frequency diagram represent time, frequency and channel, respectively, forming a more comprehensive and rich feature representation of the few-channel motor imagery electroencephalogram signal. The feature enhancement method through feature connection provides a three-dimensional image representation of the electroencephalogram signal, reducing the domain difference between the electroencephalogram signal domain and the natural image domain, so that the subsequent step of transfer learning can perform better.

[0044] Step 5: The channel-dependent multi-layer electroencephalogram time-frequency diagram is randomly divided into a training and validation set and a test set according to a ratio of 9:1, and the training and validation set is randomly divided into a training set and a validation set according to a ratio of 9:1.

[0045] Step 6: Pre-train the EfficientNet-B0 network using a large-scale image dataset ImageNet, and retain the pre-training weights of the EfficientNet-B0 network.

[0046] Step 7: Delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer electroencephalogram time-frequency diagram classification head, and generate an EfficientNet-B0 network suitable for electroencephalogram signal classification; the channel-dependent multi-layer electroencephalogram time-frequency diagram classification head includes four layers, the first layer is a global average pooling layer, the second layer is a fully connected layer using a ReLU activation function, in this embodiment, a fully connected layer with 128 neurons is used, the third layer is a Dropout layer with a dropout rate of 0.5, and the fourth layer is a fully connected layer with two neurons using softmax as the activation function, as shown in Figure 3 .

[0047] Step 8: Input the training and validation set in step 4 to the EfficientNet-B0 network suitable for electroencephalogram signal classification in step 7 for transfer learning training, and use the trained network as the final classifier; the transfer learning training process uses an Adam optimizer for training optimization; an early stopping strategy is used, and if the loss of the validation set cannot be reduced after 20 consecutive epochs, the transfer learning training is terminated.

[0048] Step 9: After the training is completed, the model is evaluated and verified using the test set, and if the evaluation indicators do not meet the standards, the accuracy is improved by increasing the original few-channel motor imagery electroencephalogram signal data in step 1, adjusting the EfficientNet-B0 network structure suitable for electroencephalogram signal classification in step 7, or further fine-tuning the transfer learning training hyperparameters in step 8, the evaluation indicators include accuracy, recall rate and F1 score. Wherein, the adjustment of the EfficientNet-B0 network structure suitable for electroencephalogram signal classification can adjust the number of neurons in the second full connection layer, if you want to improve the accuracy, you can try to increase the number of neurons based on 128, if you want to improve the model inference speed, you can reduce the number of neurons based on 128.

[0049] The channel-dependent multi-layer time-frequency graph designed by the application retains the time and frequency characteristics of motor imagery electroencephalogram signals, enhances the correlation between channels, and better represents the time, frequency and channel characteristics of electroencephalogram signals under the limitation of few channels. In existing research, transfer learning is rarely used in electroencephalogram signal recognition field, the reason is that the domain difference between electroencephalogram signal domain and natural image domain used for pre-training of backbone network is large. The new channel-dependent multi-layer time-frequency graph proposed by the application reduces the domain difference between electroencephalogram signal domain and natural image domain, so it can show better effect in transfer learning. The application has exceeded other existing few-channel motor imagery electroencephalogram signal detection methods on the few-channel motor imagery electroencephalogram public data set through experiments. Compared with 77.22% average classification accuracy achieved by M. Moufassih [5] et al. on the few-channel motor imagery electroencephalogram public data set BCI Competition IV dataset 2b [6], the average classification accuracy of the application on the few-channel motor imagery electroencephalogram public data set is improved to 80.21%.

[0050] The embodiment of the application also provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor, and realizes the final classifier of the few-channel motor imagery electroencephalogram signal recognition method based on transfer learning according to the above-mentioned embodiment, which is used for identifying few-channel motor imagery electroencephalogram signals.

[0051] The embodiment of the present application also provides a computer readable storage medium storing a computer program, wherein the computer program realizes the final classifier of the method for recognizing few-channel motor imagery electroencephalogram signals when executed by a processor of a computer. The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The program code can be executed completely on a machine, partially on a machine, partially on a machine as a separate software package and partially on a remote machine, or completely on a remote machine or server.

Claims

1. A method for recognizing few-channel motor imagery electroencephalogram signals, characterized in that Comprising the following steps: Step 1: uniformly time segment the original few-channel motor imagery electroencephalogram signal, remove the time segments where no motor imagery occurs, and obtain the few-channel motor imagery electroencephalogram signal; wherein the original few-channel motor imagery electroencephalogram signal is collected by a brain-computer interface of a motor imagery subject, and is labeled with a label of a specific classification of a motor imagery task; Step 2: filter the few-channel motor imagery electroencephalogram signal using a second-order Butterworth low-pass filter to retain signals in the frequency range of 8-30 Hz and remove artifacts; Step 3: perform continuous wavelet transform on the filtered signal to output a two-dimensional time-frequency graph of time and frequency; Step 4: feature connect the time-frequency graphs of different channels in the channel dimension to output a channel-dependent multi-layer electroencephalogram time-frequency graph; the feature connection is a vertical stacking of time-frequency graphs of different channels to output a channel-dependent multi-layer electroencephalogram time-frequency graph, which has three dimensions representing time, frequency, and channel; Step 5: divide the channel-dependent multi-layer electroencephalogram time-frequency graph into a training and validation set and a test set, and divide the training and validation set into a training set and a validation set; Step 6: pre-train the EfficientNet-B0 network using a large-scale image dataset ImageNet and retain the pre-training weights of the EfficientNet-B0 network; Step 7: delete the fully connected layer in the pre-trained EfficientNet-B0 network, add a channel-dependent multi-layer electroencephalogram time-frequency graph classification head, and generate an EfficientNet-B0 network suitable for electroencephalogram signal classification; the channel-dependent multi-layer electroencephalogram time-frequency graph classification head includes four layers, a first layer of global average pooling layer, a second layer of fully connected layer using ReLU activation function, a third layer of Dropout layer with a loss rate of 0.5, and a fourth layer of fully connected layer with two neurons using softmax as the activation function; Step 8: input the training and validation set in step 4 into the EfficientNet-B0 network suitable for electroencephalogram signal classification in step 7 for transfer learning training, and after training, use the final classifier for few-channel motor imagery electroencephalogram signal recognition.

2. The method for few-channel motor imagery electroencephalogram signal recognition according to claim 1, characterized in that Also comprising step 9, evaluating the final classifier obtained by training using the test set to verify the evaluation indicators, and if the evaluation indicators are not up to standard, improving the accuracy of the final classifier by the following methods: Increase the data of the original few-channel motor imagery electroencephalogram signal in step 1; Adjust the number of neurons in the second layer of the fully connected layer in the EfficientNet-B0 network structure suitable for electroencephalogram signal classification in step 7; Or fine-tune the transfer learning training hyperparameters in step 8.

3. The method for few-channel motor imagery electroencephalogram signal recognition according to claim 2, characterized in that In step 9, the evaluation indicators include accuracy, recall rate, and F1 score.

4. The method for identifying the few-channel motor imagery electroencephalogram signal according to claim 1 or 2, characterized in that In step 3, the continuous wavelet transform uses Morlet wavelet as the wavelet base function, and the scale range of the Morlet wavelet is set between 1 and 64, which is used to capture the frequency characteristics of the motor imagery electroencephalogram signal in the range of 8-30 Hz, and the continuous wavelet transform outputs a two-dimensional matrix after calculation, wherein one dimension represents time and the other dimension represents frequency.

5. The method for few-channel motor imagery electroencephalogram signal recognition according to claim 1 or 2, In step 8, the process of the transfer learning training is trained and optimized by using an Adam optimizer. An early stopping strategy is used, and if the loss of the validation set fails to decrease after 20 consecutive epochs, the transfer learning training is terminated.

6. An electronic device, characterized by The electronic device includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to realize the final classifier of the motor imagery electroencephalogram signal recognition method according to any one of claims 1-5, for identifying the motor imagery electroencephalogram signal.

7. A computer readable storage medium characterized by The computer readable storage medium stores at least one instruction or at least one program, and when the at least one instruction or the program is executed, the final classifier of the motor imagery electroencephalogram signal recognition method according to any one of claims 1-5 is realized.

Images