A method, device and storage medium for classifying motor imagery signals

Abstract

The application discloses a kind of motor imagery signal classification method, equipment and storage medium, method: to be classified motor imagery signal data is preprocessed;The classified motor imagery signal data after pre-processing is input into classification model, and the category of the classified motor imagery signal data is output by classification model, and classification model includes MI synchronization feature weighting module, multiscale feature extraction module, electrode position information extraction module and MI classification module, MI synchronization feature weighting module, multiscale feature extraction module and MI classification module are sequentially connected, and electrode position information extraction module is connected with MI classification module.The classification method of the application introduces synchronization information into MI classification problem, and the synchronization information of MI signal is used to guide the extraction of MI signal features.The multiscale feature extraction module and the electrode position information extraction module extract the time features of the MI signal at different scales and the position signals of each channel, respectively, and the model has high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer signal processing, and relates to a motor imagery signal classification method and device and a storage medium, in particular to a method capable of achieving high-precision and accurate classification of EEG motor imagery signals and application thereof. BACKGROUND

[0002] At present, the application of the brain-computer interface (BCI, Brain-Computer Interface) system based on motor imagery is mainly in the medical field, which provides a new control method for patients with movement disorders, helping them to achieve self-care in life or even complete recovery from the disease. These patients with movement disorders include brain stem injury, cerebral apoplexy, muscle atrophy, etc., who have complete brains but cannot control their bodies due to damage to peripheral nerves. The BCI system based on motor imagery (MI, motor imagery) can realize the control of external devices and replace trunk movement. In addition, the combination of motor imagery and physical therapy is an effective method for stroke rehabilitation.

[0003] So far, many studies have tried to manually extract features and then train a machine learning-based classifier to identify MI signals. Among them, researchers have proposed a popular spatial feature extraction method called common spatial pattern (CSP, Common Spatial Pattern), which greatly improves the classification performance of MI signals. CSP is suitable for two-class classification, and its basic idea is to find a spatial filter that maximizes the change between the two classes by solving a matrix decomposition problem. Due to its stable performance, researchers have proposed many improved versions of CSP. For example, filter bank common spatial pattern (FBCSP, Filter Bank Common Spatial Pattern) improves the MI classification accuracy by optimizing the subject-specific frequency band; separable common spatio-spectral patterns (SCSSP, Separable Common Spatio-spectral Patterns) consider the spectral features of MI signals, which reduces the computational complexity compared with CSP. However, these traditional methods are difficult to meet the requirements of high classification accuracy of BCI systems. It is necessary to develop new algorithms and frameworks, which are expected to be applicable to multiple disciplines at the same time.

[0004] Deep Neural Network (DNN) as a subclass of machine learning has made remarkable progress in computer vision and natural language processing. As an end-to-end model, DNN is very suitable for EEG-based MI classification. However, the traditional DNN cannot fully extract the three-dimensional position features of the brain electrode, and in addition, the classification accuracy of the DNN based only on the time domain features of the MI signal is difficult to further improve.

[0005] Therefore, it is of great practical significance to develop a method capable of realizing high-precision and accurate classification of motor imagery signals of EEG. SUMMARY

[0006] Since the prior art has the above defects, the present application provides a method capable of realizing high-precision and accurate classification of motor imagery signals of EEG, to overcome the problems of poor accuracy of traditional MI classification method, and the inability of existing DNN to fully extract the three-dimensional position features of the brain electrode and poor classification accuracy of time domain features in EEG-based MI classification.

[0007] In order to achieve the above purpose, the present application provides the following technical solutions:

[0008] A classification method of motor imagery signals, comprising the following steps:

[0009] (1) pre-processing the motor imagery signal data to be classified;

[0010] (2) inputting the pre-processed motor imagery signal data to be classified into a classification model, and the classification model outputs the category of the motor imagery signal data to be classified;

[0011] The classification model comprises an MI synchronous feature weighting module, a multi-scale feature extraction module, an electrode position information extraction module and an MI classification module, the MI synchronous feature weighting module, the multi-scale feature extraction module and the MI classification module are sequentially connected, and the electrode position information extraction module is connected with the MI classification module.

[0012] The training process of the classification model is a process of constantly adjusting the model parameters with the motor imagery signal data in the training data set as the input and the known category information corresponding to the motor imagery signal data as the theoretical output.

[0013] The motion imagination signal classification method of the application introduces synchronization information into the MI classification problem when constructing the classification model, guides the feature extraction of the MI signal by using the synchronization information of the MI signal, the multi-scale feature extraction module in the classification model can extract the features of the MI signal in different scales, the electrode position information extraction module can extract the position signals of each channel of the MI signal, the information categories extracted by the model are enriched, the extracted information is combined with the synchronization information (the spatial and temporal features are fully extracted) and classified together, which can greatly improve the accuracy of the classification model and has good application prospect.

[0014] As a preferred technical solution:

[0015] The MI synchronization feature weighting module extracts corresponding information from the MI signal and the SL data by the same convolution operation, and then weights the original MI signal according to the corresponding position based on the synchronization likelihood index.

[0016] The multi-scale feature extraction module extracts features from the MI signal processed by the MI synchronization feature weighting module by using multiple convolutions in different time dimensions, and then splices in the time dimension.

[0017] The electrode position information extraction module includes multiple graph convolution neural networks (GCN) connected in sequence.

[0018] The spatial features extracted by the taking module are convolved, flattened, and classified.

[0019] The application further provides a computer device, which comprises:

[0020] at least one processor; and

[0021] a memory in communication connection with the at least one processor; wherein

[0022] The memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the motion imagination signal classification method.

[0023] In addition, the application further provides a computer readable storage medium, which stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the motion imagination signal classification method.

[0024] The above technical solution is only one feasible technical solution of the present application, and the protection scope of the present application is not limited to this.

[0025] The above-mentioned application has the following advantages or beneficial effects:

[0026] (1) The motor imagery signal classification method of the present application introduces synchronization information into the MI classification problem when constructing the classification model, and uses the synchronization information of the MI signal to guide the feature extraction of the MI signal.

[0027] (2) The motor imagery signal classification method of the present application can extract the features of the MI signal at different scales through the multi-scale feature extraction module in the classification model.

[0028] (3) The motor imagery signal classification method of the present application can extract the position signals of each channel of the MI signal through the electrode position information extraction module, enriching the information categories extracted by the model.

[0029] (4) The motor imagery signal classification method of the present application can greatly improve the accuracy of the classification model by extracting the position information and synchronization information of the MI signal (realizing sufficient extraction of spatial and temporal features) and classifying them together, which has great application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0030] The present application and its features, shapes and advantages will become more apparent through reading the detailed description of the non-limiting embodiments with reference to the following drawings. The same reference signs indicate the same parts throughout the drawings. The drawings are not necessarily drawn to scale, the emphasis being on illustrating the main idea of the present application.

[0031] Figure 1 Flow chart of the motor imagery signal classification method in Example 1;

[0032] Figure 2 Structure diagram of the motor imagery signal classification model constructed for Example 1;

[0033] Figure 3 Structure diagram of the MI synchronization feature weighting module;

[0034] Figure 4 Structure diagram of the multi-scale feature extraction module;

[0035] Figure 5 Structure diagram of the electrode position information extraction module;

[0036] Figure 6 Structure diagram of the classification module. DETAILED DESCRIPTION

[0037] The structure of the present application will be further described below in combination with the drawings and specific examples, but not as a limitation of the present application.

[0038] Example 1

[0039] A method for classifying motor imagery signals, the steps of which are specifically as shown Figure 1

[0040] (1) Obtaining the training data set:

[0041] Step 1.1, data acquisition of the subject to form the laboratory data set, and pre-processing of the laboratory data set:

[0042] 1.1.1: Obtain the laboratory data set. The method of collecting the laboratory data set is exactly the same as the standard data set BCI Competition IV-2a data set. The data set uses 22 Ag / AgCl electrodes to record twice of each subject, a total of 9 subjects. For each subject, the two experiments are carried out within two days. During the experiment, the subject is comfortably seated in front of the screen. At the beginning of each test, a cross with a short sound warning will appear on the black screen. After 2 seconds, an arrow pointing left, right, down or up (corresponding to left hand, right hand, foot or tongue) appears, indicating the performance of the motor imagery task. The subject is required to perform the MI task from t=2s to t=6s. Each subject contains 72 tests, and a total of 288 tests are obtained from each subject. The EEG signal used in each test can be expressed as X∈R C×T , where C=22 is the number of channels, T=1000 is the number of data points;

[0043] 1.1.2: Preprocessing of the laboratory data set. The preprocessing method of the laboratory data set is exactly the same as the standard data set. The laboratory data set is sampled at 250Hz, and band-pass filtered between 0.5Hz and 100Hz, then a 50Hz notch filter is used to suppress noise, and finally the laboratory data set is divided into training data set and test data set according to the proportion of the standard data set.

[0044] Step 1.2, use MATLAB to write program to extract the synchronous features (i.e. time features) and normalized mutual information (i.e. spatial features-electrode position information) of the MI signals of the training data set;

[0045] ​1.2.1: Extracting the synchronization feature of the MI signal. This scheme uses the Synchronous Likelihood (SL) index to represent the synchronization feature. The theoretical derivation is as follows: SL is used to measure the nonlinear synchronization between each pair of electroencephalogram signals. The degree of synchronization depends on the similarity of the time series in shape, which is measured by the closeness of the points in the corresponding trajectory obtained after phase space reconstruction. Given each length of the simultaneously recorded electroencephalogram signals, reconstruct the embedding vector according to the Takens embedding theorem:

[0046] X k,j =(x k,j ,x k,j+l ,x k,j+2l ,...,x k,j+(m-1)l )k=1,2,...,M,i=1,2,...,N

[0047] where l is the lag and m is the embedding dimension.

[0048] For each channel signal k and each time point i, we define as the probability that the embedding vector distance is less than ε:

[0049]

[0050] where |·| is the Euclidean distance, φ is the Heaviside step function, and ω1 and ω2 are the widths of the two time windows.

[0051] For each time series k, the critical distance ε k,j is determined by setting where p ref <<1. Then, for each discrete time pair (i, j) within the considered window, the number of channels H i,j can be determined, where the distance between the embedding vectors X k,i and X k,j is less than ε k,i :

[0052]

[0053] This number is between 0 and M, reflecting how many embedding signals are "similar" to each other.

[0054] The synchronization likelihood S k,i,j for each channel k and each discrete time pair (i, j) can be defined as

[0055]

[0056] By averaging over all channels j, the synchronization likelihood S k,iSL describes the intensity of the channel in time synchronized with all other channels as follows:

[0057]

[0058] As the synchronization between a pair of signals increases, S k,i increases, and is 1 when they are perfectly synchronized and 0 when perfectly unsynchronized;

[0059] 1.2.2: Extracting the Normalized Mutual Information (NorMI) of the MI signals. This scheme uses NorMI to represent the electrode position information. The theoretical derivation is as follows: Normalized Mutual Information quantifies the amount of information that two signals share with each other. Let ρ(x) = P r {X = x} be the probability density function of the random variable X, and ρ(y) = P r {Y = y} be the probability density function of the random variable Y. The joint probability density is defined as ρ(x, y) = P r {X = x, Y = y}. Shannon entropies H(X) and H(Y) measure the average information obtained from the observation of random variables X and Y. They are defined as

[0060] H(X) = -∑ x ρ(x) log(ρ(x)) H(Y) = -∑ y ρ(y) log(ρ(y))

[0061] The joint entropy H(X, Y) is

[0062] H(X, Y) = -∑ x ∑ y ρ(x, y) log(ρ(x, y))

[0063] The conditional entropy of X given Y is defined as:

[0064] H(X|Y) = -∑ x ∑ y ρ(x, y) log(ρ(x|y))

[0065] where ρ(x|y) = P r {X = x|Y = y} is the conditional probability. The form of the conditional entropy is similar to that of the Shannon entropy. The conditional entropy can be expressed as H(X, Y) = H(X|Y) + H(Y).

[0066] The mutual information I(X; Y) measures the amount of information shared between X and Y:

[0067] I(X; Y) = ∑ x ∑ yp(x, y) log(p(x, y) / p(x)p(y))

[0068] It can also be expressed as:

[0069] I(X; Y) = H(X, Y) - H(X|Y) - H(Y|X)

[0070] Mutual information needs to be normalized to measure the connectivity between variables:

[0071]

[0072] The normalized mutual information between two electrodes can be used to measure the functional connection strength between these electrodes. Therefore, the graph structure describing the connectivity of the brain is defined as follows: the nodes are the channels, and the edges are the normalized mutual information between the nodes.

[0073] (2) Construction of classification model:

[0074] Step 2.1, construct the MI synchronous feature weighting module and the multi-scale feature extraction module:

[0075] The overall structure of the framework is shown in Figure 2 The framework is divided into four parts: the MI synchronous feature weighting module, the multi-scale feature extraction module, the electrode position information extraction module, and the classification module. The MI synchronous feature weighting module and the multi-scale feature extraction module obtain the time domain features of the MI signal; the electrode position information extraction module obtains the position features of the multi-channel MI signal.

[0076] 2.1.1: The MI synchronous feature weighting module is shown in Figure 3 The MI synchronous feature weighting module first extracts the corresponding information of the MI signal and the SL data with the same convolution operation, and at the same time, the synchronous likelihood index is used to weight the original MI signal according to the corresponding position. The specific process is shown in the following formula:

[0077]

[0078] where X k,i is the MI signal segment of the i-th time period of the k-th channel, is the weighted result, is to avoid the complete loss of corresponding information when it is 0. The final processing result of the MI synchronous feature weighting module is X1∈R N×C×T / 4 .

[0079] 2.1.2: The multi-scale feature extraction module processes the MI signal processing result X1∈R N ×C×T / 4The processing is performed. The multi-scale feature extraction module uses three different time dimensions of convolution for feature extraction, and splicing in the time dimension, the specific process is shown as follows:

[0080]

[0081] F = concat (F1, F2, F3) ∈ R N×C×T′

[0082] Where Lsconv, Msconv and Ssconv represent large-scale convolution, medium-scale convolution and small-scale convolution respectively, concat(·) is a splicing function in the time dimension, and T' = T1 + T2 + T3.

[0083] After the above operation, the multi-scale feature extraction module combines each channel using convolution operation. The processing result generated by the final multi-scale feature extraction module is X2 ∈ R N×T' .

[0084] Step 2.2, constructing an electrode position information extraction module:

[0085] The electrode position information extraction module first constructs a graph network for the MI signal through NorMI. In order to make the graph network sparse and improve the network training speed. The adjacency matrix A of the graph network is defined as follows:

[0086]

[0087] Where τ is an artificial threshold for making the adjacency matrix sparse.

[0088] The Laplacian matrix L is further calculated using the adjacency matrix A, as follows:

[0089]

[0090] Where I is the identity matrix, and D is the degree matrix of the adjacency matrix A. The GConv update formula of each layer can be defined as:

[0091]

[0092] Where θ ∈ R T×G is the filter parameter matrix of GConv, σ is the ReLU activation function, and G is the number of nodes after GConv. Therefore, the spatial feature S3 can be obtained by three GConv of the original MI signal X. The original MI signal conversion process is Where G1, G2, G3 are the operations of GConv.

[0093] Step 2.3, constructing an MI classification module:

[0094] In the classification module, firstly, the time and space features extracted by the model are fused. Specifically, the time feature X2 and the space feature X3 are connected in the channel dimension by the following formula to obtain the discriminative feature Z:

[0095] Z = concat(X3, X4) e R 2N×T′

[0096] Then the classification module converts Z into Z using a convolution operation. * e R N×T' / 2 Finally, after the Z is flattened, it is input into the classifier, and the softmax function is used for classification. *

[0097] (3) Training the classification model and the comparative model (CSP, CSSP, FBCSP, SCSSP, EEGNet):

[0098] Specifically, the motor imagery signal data in the training data set is taken as the input, and the known category information corresponding to the motor imagery signal data is taken as the theoretical output, and the parameters of the classification model and the comparative model are adjusted to complete the training.

[0099] (4) Preprocessing the motor imagery signal data to be classified (selected from the test data set in step 1.1) (same as step 1.1.2), inputting the preprocessed motor imagery signal data to be classified into the classification model / comparative model, and the classification model / comparative model outputs the category to which the motor imagery signal data to be classified belongs.

[0100] The test result is that the classification accuracy of the classification model of the present application is nearly 10% higher than that of the EEGNet model with the best classification performance in the above comparative models. After testing, it is found that the classification model of the present application has better accuracy than CSP, CSSP, FBCSP, SCSSP and EEGNet.

[0101] Those skilled in the art should understand that those skilled in the art can realize variations in combination with the prior art and the above embodiments, which are not described here. Such variations do not affect the essential content of the present application, and are not described here.

[0102] ​The preferred embodiments of the present application have been described. It is to be understood that the application is not limited to the above specific embodiments, and that devices and structures not described in detail should be understood to be implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the technical solutions of the present application, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present application, which does not affect the essential content of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the content of the technical solutions of the present application, still belongs to the scope of protection of the technical solutions of the present application.

Claims

1. A method of classifying a motor imagery signal, characterized by: The method comprises the following steps: (1) preprocessing the motor imagery signal data to be classified; (2) inputting the preprocessed motor imagery signal data to be classified into a classification model, and the classification model outputs the category to which the motor imagery signal data to be classified belongs; The classification model comprises an MI synchronization feature weighting module, a multi-scale feature extraction module, an electrode position information extraction module, and an MI classification module, the MI synchronization feature weighting module, the multi-scale feature extraction module, and the MI classification module are sequentially connected, and the electrode position information extraction module is connected with the MI classification module; The training process of the classification model is a process of constantly adjusting model parameters by taking the motor imagery signal data in the training data set as input and taking the known category information corresponding to the motor imagery signal data as theoretical output; The MI synchronization feature weighting module extracts corresponding information from the MI signal and the SL data by the same convolution operation, and then weights the original MI signal according to the corresponding position, and the SL data is a synchronization likelihood index; The electrode position information is the normalized mutual information of the MI signal.

2. The method of classifying motor imagery signals according to claim 1, wherein, The multi-scale feature extraction module extracts features from the MI signal processed by the MI synchronization feature weighting module by using multiple convolutions in different time dimensions, and then splices in the time dimension.

3. The method of classifying motor imagery signals according to claim 2, wherein, The electrode position information extraction module comprises multiple graph convolutional neural networks connected in sequence.

4. The method of classifying motor imagery signals according to claim 3, wherein, The MI classification module convolves, flattens, and classifies after fusing the time features extracted by the multi-scale feature extraction module and the spatial features extracted by the electrode position information extraction module.

5. A computer device, characterized by: The computer device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the motor imagery signal classification method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the motor imagery signal classification method according to any one of claims 1-4.

Images