EEG Signal Classification Method Based on Multi - segment Signal Random Recombination and Interactive Bidirectional RNN

Abstract

The present invention discloses an EEG signal classification method based on multi-segment signal random recombination and interactive bidirectional RNN. First, the collected original EEG data is segmented, and then the signals of each category are randomly sliced in turn, and their lengths are kept at 70% to 90% of the original. Subsequently, the segments of multiple tasks are recombined together, and this processing process is called "multi-segment signal random slicing recombination (McRFS)"; in each iteration, the fully connected layer of the model is used to extract the activated brain regions, thereby reducing the dimension of complex data; segments with random lengths and originating from random positions are fed into the IBiRNN network to further fuse forward and backward features; finally, the Softmax layer is used to activate the features to obtain the classification result, realizing the distinction of different brain states. The present invention can effectively improve the classification performance of EEG signals and solve the task of distinguishing brain states to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of signal processing, and particularly relates to an EEG signal classification method based on multi-clip random recombination and interactive bidirectional RNN; Background Art

[0002] Electroencephalography (EEG) has been widely used in the fields of brain-computer interface and neuroscience technology due to its non-invasive and low-cost characteristics. Among them, distinguishing brain states is an important topic for understanding the working mechanism of the brain. Based on this, the brain-computer system can capture more valuable features, so as to achieve more accurate recognition or control. To achieve this goal, researchers have designed and proposed many models and methods, mainly including machine learning and deep learning algorithms. In existing work, algorithms based on recurrent neural network (RNN) have been widely used in the classification research of brain states due to their sensitivity to time series and efficient learning ability. However, there is still much room for improvement in the current classification performance, and there are still some problems hindering researchers from understanding the working mechanism of the brain. On the one hand, algorithms based on RNN rarely improve the architecture of neural networks in combination with the characteristics of EEG signals. On the other hand, how to slice and combine the original EEG signals is also an important factor affecting the performance of the model. More and more work shows that compared with using the entire time series of EEG signals to distinguish brain states, studying its partial segments may be more effective. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides an EEG signal classification method McRFS-IBiRNN (multi-clip random fragment strategy-based interactive bidirectional recurrent neural network) based on multi-clip random recombination and interactive bidirectional RNN. First, the collected original EEG data is segmented, and then each type of signal is randomly sliced in turn, and its length is kept at 70% to 90% of the original. Subsequently, the segments of multiple tasks are recombined together, and this processing process is called "multi-clip random fragment strategy (McRFS)"; in each iteration, the fully connected layer of the model is used to extract the activated brain regions, so as to reduce the dimension of complex data; the segments with random lengths and originating from random positions are fed into the IBiRNN network to further fuse forward and backward features; finally, the Softmax layer is used to activate the features to obtain the classification result, realizing the distinction of different brain states. The present invention can effectively improve the classification performance of EEG signals and solve the task of distinguishing brain states to a certain extent.

[0004] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0005] Step 1: Preprocessing of electroencephalogram signals;

[0006] Data preprocessing is performed for each subject, and each subject contains multiple types of tasks;

[0007] Remove the data contaminated by noise in the original electroencephalogram signals in the electroencephalogram dataset, and transpose and preprocess the remaining data to obtain a signal matrix for each type of task of a single subject; then, split the signal matrix into multiple data segments; splice the data segments of all types of tasks of each subject in the time dimension to form a spliced data segment, and then randomly select most of the spliced data segments as the training set and the remaining small part of the spliced data segments as the test set;

[0008] Step 2: Random slicing and recombination of multi-segment signals;

[0009] Randomly slice each spliced data segment in the training set at a random position from each segment corresponding to each type of task, and keep its length within 70%-90% of the original sequence length; then recombine the sliced data segments of each spliced data segment together to form a new data segment;

[0010] Step 3: Construct an electroencephalogram signal classification model using an interactive bidirectional RNN;

[0011] Step 3-1: Define a set of signal sequences {x 1 ,…,x m} input to the recurrent layer, and the output calculation formula at each time point t is:

[0012] h t =tanh(Uh t-1 +Wx t +b) (1)

[0013] where U and W respectively represent the weight matrices of the hidden layer and the input features, b is the bias term, and h t represents the hidden layer state at time t, t = 1,…,m;

[0014] Step 3-2: For the new data segment obtained in Step 2, first pass through two GRU recurrent layers, denoted as F1 and F2 respectively. The first hidden state of the GRU is defined as follows:

[0015]

[0016] where ⊙ represents element-wise multiplication of matrices, z t is the update gate activation, r tis the reset gate activation, σ is the sigmoid function; h t-1 represents the hidden layer state at time t-1, represents the candidate hidden layer state, U z , U r , U h and W z , W r , W h respectively represent the weight matrices of the corresponding hidden layer and input features, b z , b r , b h represents the corresponding bias term;

[0017] The neurons in the recurrent layer F1 process the signals sequentially according to the time order of the input sequence, combining the forward state, and then feed the calculation results back to the next neuron in the current recurrent layer and the corresponding neuron in the recurrent layer F2. Repeat the above steps for all neurons. After two layers of forward processing, the final state is obtained

[0018] The structures of the backward recurrent layers B1 and B2 are the same as those of the recurrent layer. The difference is that the states of the two layers in the backward recurrent layer are read from the end of the sequence and propagated from back to front to obtain the final output

[0019] Step 3-3: The interactive bidirectional RNN adds a concatenation function between the B1 / F1 layer and the B2 / F2 layer, concatenates the outputs of the recurrent layer F1 and the backward recurrent layer B1, and feeds them to the recurrent layer F2 and the backward recurrent layer B2; both the recurrent layer F2 and the backward recurrent layer B2 receive the same input and have an explicit connection; the calculation formula is as follows:

[0020]

[0021] where is the concatenation of the forward state and the backward state of the same size; represents the GRU output of the F1 layer, represents the GRU output of the B1 layer, represents the GRU output of the F2 layer, represents the GRU output of the B2 layer;

[0022] Step 4: Deep sparse penalty;

[0023] During the training process, the optimization objective is as follows:

[0024]

[0025] where is the global error loss, defined as the cross-entropy between the model output and the true labels; ‖O Full ‖ represents the sparsity penalty of the output matrix of the fully connected layer, expressed using the L1 norm, and β is used to adjust the weight of the sparsity penalty term; ‖W Full ‖ is the sparsity penalty of the weight matrix of the fully connected layer, and λ is used as the weight parameter of this penalty term to control sparsity;

[0026] Step 5: After the training set and the test set are processed in Step 2, the EEG signal classification model constructed in Step 3 is trained and tested to obtain the final EEG signal classification model McRFS-IBiRNN based on random recombination of multiple segments of signals and interactive bidirectional RNN. The output of McRFS-IBiRNN is the final classification result.

[0027] Preferably, the EEG dataset is the motor / imagery EEG dataset provided by PhysioNet.

[0028] The beneficial effects of the present invention are as follows:

[0029] 1. The present invention uses the method of randomly recombining multiple segments of signals to streamline redundant data, effectively increasing the classification performance of the model while improving the diversity of the data structure;

[0030] 2. The sparse penalty algorithm of the present invention enables the fully connected layer to pay more attention to the significantly activated brain regions, enabling the model to extract the key features of the data;

[0031] 3. The interactive bidirectional RNN of the present invention deeply fuses the forward and backward representations during the iteration process, effectively improving the timeliness and generalization performance of classification. Description of the Drawings

[0032] Figure 1 is the overall structural schematic diagram of the McRFS-IBiRNN model of the present invention.

[0033] Figure 2 is the structural schematic diagram of the interactive stacked BiRNN of the present invention. Detailed Embodiments

[0034] The present invention will be further described below in conjunction with the drawings and embodiments.

[0035] As Figure 1 shown, the EEG signal classification method based on random recombination of multiple segments of signals and interactive bidirectional RNN includes the following steps:

[0036] Step 1: Preprocessing of EEG signals;

[0037] The model performs data preprocessing for each subject, such as Figure 1(a) As shown in the Multi-clip RFS section, taking the Movement / Imagery EEG dataset provided by PhysioNet as an example, each subject in it contains four types of tasks: opening and closing the left and right fists (TASK1), imagining opening and closing the left and right fists (TASK2), opening and closing both fists or feet (TASK3), imagining opening and closing both fists or feet (TASK4), corresponding to the four dimensions of the input matrix TASK1-4 in the Multi-clip RFS section. First, remove the data segments contaminated by noise in the original EEG signals of each task, and transpose and preprocess the data to obtain a signal matrix of 19680 time steps × 64 channels for each task of a single subject; then, segment the signal matrix, and slice the 123-second (19680 time steps) signal with a time step of about 1 s (164 time steps, that is, 164 / 160 = 1.025 s) to obtain 120 data segments of 164 time steps × 64 channels. Finally, splice the 4 types of movement / imagery tasks of each subject in the time dimension to obtain 120 data segments with a dimension of 656 time steps × 64 channels for subsequent random slicing and recombination of multi-segment signals.

[0038] Step 2: Random slicing and recombination of multi-segment signals;

[0039] As Figure 1 (a) As shown on the right side of the Multi-clip RFS section, for the 120 data segments of 164 time steps × 64 channels × 4 tasks obtained in Step 1, randomly slice with a random length from a random position in the corresponding segments of each task in turn, so that the length of the obtained new sequence remains 70%-90% of the original sequence length, and then recombine the segments of the 4 tasks together to obtain 120 task data segments of 64 channels × 4 but with an indefinite time length. Use McRFS to process the data. Since the slicing operation contains both randomness of the slicing position and randomness of the slicing length, this can enrich the diversity of the data. At the same time, in the obtained new sequence, the data is continuous in the time dimension, which can ensure that the model can learn the temporal correlation stored in the EEG signals. So far, the operations of data processing and slicing and recombination have been completed. In subsequent experiments, randomly select 80 segments as the training set and the remaining 40 segments as the test set.

[0040] Step 3: Interactive bidirectional RNN;

[0041] In each iteration, the fully connected layer of the model is used to extract the activated brain regions, thereby reducing the dimension of complex data; segments with a random length and from a random position are fed into the interactive bidirectional RNN network we proposed to further fuse the forward and backward features; Figure 2Shows in detail Figure 1 (b) The partial model structure of Interactive BiRNN. In the interactive bidirectional RNN, the forward layer has two layers, including F1 and F2, and the backward layer consists of B1 and B2. A set of signal sequences {x 1 ,…,x m} input into the recurrent layer, and the output calculation formula at each time point t is:

[0042] h t =tanh(Uh t-1 +Wx t +b) (1)

[0043] Among them, U and W respectively represent the weight matrices of the hidden layer and input features, b is the bias term, and h t represents the hidden layer state at time t.

[0044] For the data processed in step 2, it first passes through two layers of GRU (gated recurrent unit) recurrent layers, denoted as F1 and F2 respectively. The first-layer hidden state of GRU is defined as follows:

[0045]

[0046] Among them, ⊙ represents element-wise multiplication of matrices, z t is the update gate activation, and r t is the reset gate activation.

[0047] The neurons in the first layer process the signals sequentially according to the time order of the input sequence, combining the forward state, and then feedback the calculation results to the next neuron in the current layer and the corresponding neuron in the second layer. Repeat the above steps for all neurons. After two layers of forward processing, the final state can be obtained The backward recurrent layers B1 and B2 have a similar structure. The difference is that the states of these two layers are read from the end of the sequence and propagated from back to front to obtain the final output An additional concatenation function is added between the first layer and the second layer in the interactive bidirectional RNN to concatenate the outputs of the forward and backward units and send them to the next forward layer and backward layer. Both F2 and B2 receive the same input, and there is an explicit connection between the forward layer and the backward layer. The calculation formula is as follows:

[0048]

[0049] Among them is the concatenation of the forward state and the backward state of the same size, corresponding to the "interactive connection" module we proposed.

[0050] Step 4: Deep Sparse Penalty;

[0051] During the model training process, we propose to use the sparse penalty algorithm to help the fully connected layer filter out low-correlated brain regions and extract significantly activated brain regions. The specific optimization objective function is as follows:

[0052]

[0053] where is the global error loss, which is defined as the cross-entropy between the model output and the true label in the present invention; ‖O Full ‖ represents the sparsity penalty of the fully connected layer output matrix, which is represented by the L1 norm in the present invention, and β is used to adjust the weight of the sparse penalty term; ‖W Full ‖ is the sparsity penalty of the fully connected layer weight matrix, and λ is used as the weight parameter of this penalty term to control the sparsity.

[0054] Step 5: After preprocessing the dataset through Step 1, input it into Steps 2-4 to obtain the final McRFS-IBiRNN model, and the output is the final classification result. Specific Embodiment:

[0056] 1. EEG Signal Preprocessing

[0057] The model performs data preprocessing for each subject. As Figure 1 (a) shown in the Multi-clip RFS part, taking the Movement / Imagery EEG dataset provided by PhysioNet as an example, each subject in it contains four types of tasks: opening and closing the left and right fists (TASK1), imagining opening and closing the left and right fists (TASK2), opening and closing both fists or feet (TASK3), imagining opening and closing both fists or feet (TASK4), corresponding to the four dimensions of the input matrix TASK1-4 in the Multi-clip RFS part. First, remove the data segments contaminated by noise in the original EEG signals of each task, and transpose and preprocess the data to obtain a signal matrix of 19680 time steps × 64 channels for each task of a single subject; then, segment the signal matrix, and slice the 123-second (19680 time steps) signal with a time step of about 1 s (164 time steps, that is, 164 / 160 = 1.025 s) to obtain 120 data segments of 164 time steps × 64 channels. Finally, splice the 4 types of movement / imagery tasks of each subject in the time dimension to obtain 120 data segments with a dimension of 656 time steps × 64 channels for subsequent random slicing and recombination of multi-segment signals.

[0058] 2. Random Slicing and Recombination of Multi-segment Signals

[0059] As Figure 1 (a) As shown on the right side of the Multi-clip RFS part, for the 120 164-time-step × 64-channel × 4-task data segments obtained in step 1, random-length slices are sequentially taken from random positions of the segments corresponding to each task, so that the length of the obtained new sequences remains between 70% and 90% of the original sequence length. Subsequently, the segments of the 4 tasks are recombined to obtain 120 64-channel × 4 but variable-time-length task data segments. The McRFS is used to process the data. Since the slicing operation includes both randomness in the slicing position and randomness in the slicing length, this can enrich the diversity of the data. At the same time, in the obtained new sequences, the data is continuous in the time dimension, which can ensure that the model can learn the temporal correlation stored in the EEG signals. So far, the data processing and slicing recombination operations have ended. In the subsequent experiments, 80 segments are randomly selected as the training set, and the remaining 40 segments are used as the test set.

[0060] 3. Interactive Bidirectional RNN

[0061] In each iteration, the fully connected layer of the model is used to extract the activated brain regions, thereby reducing the dimension of the complex data; segments with random lengths and from random positions are fed into the interactive bidirectional RNN network we proposed to further fuse the forward and backward features; Figure 2 is shown in detail Figure 1 (b) The model structure of the Interactive BiRNN part. In the interactive bidirectional RNN, the forward layer has two layers, including F1 and F2, and the backward layer consists of B1 and B2. A set of signal sequences {x 1 , …, x m} input to the recurrent layer, and the output calculation formula at each time point t is:

[0062] h t = tanh(Uh t-1 + Wx t + b) (1)

[0063] where U and W respectively represent the weight matrices of the hidden layer and the input features, b is the bias term, and h t represents the hidden layer state at time t.

[0064] For the data processed in step 2, it first passes through two layers of GRU (gated recurrent unit) recurrent layers, denoted as F1 and F2 respectively. The first hidden state of the GRU is defined as follows:

[0065]

[0066] where ⊙ represents element-wise multiplication of matrices, zt is update gate activation, r t is reset gate activation.

[0067] The first layer of neurons processes the signals sequentially according to the time order of the input sequence, combines the forward state, and then feeds the calculation results back to the next neuron in the current layer and the corresponding neuron in the second layer. Repeat the above steps for all neurons. After two layers of forward processing, the final state can be obtained. The reverse recurrent layers B1 and B2 have a similar structure. The difference is that the states of these two layers are read from the end of the sequence and propagated from back to front to obtain the final output. The interactive bidirectional RNN adds a concatenation function between the first layer and the second layer, concatenates the outputs of the forward and backward units together, and feeds them to the next forward layer and backward layer. Both F2 and B2 receive the same input, and there is an explicit connection between the forward layer and the backward layer. The calculation formula is as follows:

[0068]

[0069] where is the concatenation of the forward state of the same size and the backward state , corresponding to the "interactive connection" module we proposed.

[0070] 4. Deep Sparse Penalty

[0071] During the model training process, we propose to use the sparse penalty algorithm to help the fully connected layer filter out low-correlated brain regions and extract significantly activated brain regions. The specific optimization objective function is as follows:

[0072]

[0073] where is the global error loss, which is defined as the cross-entropy between the model output and the true label in the present invention; ‖O Full ‖ represents the sparsity penalty of the fully connected layer output matrix, which is represented by the L1 norm in the present invention, and β is used to adjust the weight of the sparse penalty term; ‖W Full ‖ is the sparsity penalty of the fully connected layer weight matrix, and λ is used as the weight parameter of this penalty term to control the sparsity.

[0074] 5. Testing Phase

[0075] After preprocessing the dataset through Step 1, it is input into Step 2 and Step 3 to obtain the output of the McRFS-IBiRNN model, which is the final classification result. In the four-class classification task of the Movement / Imagery dataset, the subject-level accuracy of 96.97% and the group-level four-class classification accuracy of 99.34% are obtained respectively. When using the model to test new subjects, an accuracy of 99.01% can be achieved. We use the GigaDB dataset provided by Handong International University to verify the generalization of the proposed model. The model still maintains extremely impressive classification performance on this dataset, reaching the binary classification accuracies of 99.71% and 98.01% at the subject level and group level respectively, as well as the four-class classification accuracies of 97.63% and 98.66%, which strongly proves the powerful generalization and reproduction ability of McRFS-IBiRNN.

[0076] The present invention proposes an EEG signal classification method based on multi-segment signal random recombination and interactive bidirectional RNN, proving that using multi-segment signal random recombination (McRFS) can better organize EEG input signals, and using interactive bidirectional RNN to fuse the forward and backward features of EEG signals to obtain temporal correlation for EEG classification has high performance, which is a pioneering research in distinguishing brain states. In addition, the designed deep sparse penalty algorithm significantly improves the robustness and adaptability of the model to cope with huge inter-individual differences. On the one hand, since directly analyzing the entire time series of EEG signals may obtain poor classification performance, to solve this problem, the McRFS strategy streamlines redundant data while increasing the diversity of data structures, generating more suitable inputs for the RNN model and achieving better performance compared with ordinary RFS and multi-segment RFS, proving the importance of organizing input EEG signals. On the other hand, the proposed interactive BiRNN can better learn bidirectional information and deeply fuse the extracted features compared with ordinary RNN, so as to efficiently obtain more information with limited resources, which can greatly improve the classification performance. Generally speaking, the proposed McRFS-IBiRNN model is novel, effective and stable, and McRFS-IBiRNN shows powerful capabilities and advantages in distinguishing brain states and EEG signal classification.

Claims

1. A method for classifying EEG signals based on multi-segment signal random recombination and interactive bidirectional RNN, characterized in that It includes the following steps: Step 1: Preprocessing of EEG signals; Data preprocessing is performed for each subject, and each subject contains multiple types of tasks; Remove the data contaminated by noise in the original EEG signals in the EEG dataset, and transpose and preprocess the remaining data to obtain the signal matrix of each type of task for a single subject; then, divide the signal matrix into multiple data segments; splice the data segments of all types of tasks of each subject in the time dimension to form a spliced data segment, and then randomly select most of the spliced data segments as the training set and the remaining small part of the spliced data segments as the test set; Step 2: Random slicing and recombination of multi-segment signals; Randomly slice each spliced data segment in the training set at a random position of the segment corresponding to each type of task, and keep its length within 70%-90% of the original sequence length; then recombine the data segments after slicing each spliced data segment together to form a new data segment; Step 3: Construct an EEG signal classification model using an interactive bidirectional RNN; Step 3-1: Define a set of signal sequences input to the recurrent layer , at each time point , the output calculation formula is: Among them, respectively represent the weight matrices of the hidden layer and the input features, is the bias term, represents the hidden layer state at time ; Step 3-2: For the new data segment obtained in Step 2, first pass through two GRU recurrent layers, denoted as F1 and F2 respectively. The first hidden state of the GRU is defined as follows: where denotes element-wise multiplication of matrices, is the update gate activation, is the reset gate activation, is the sigmoid function; denotes the hidden layer state at time denotes the candidate hidden layer state, and denote the weight matrices of the corresponding hidden layer and input features respectively, denotes the corresponding bias term; The neurons in the recurrent layer F1 process the signals sequentially according to the time order of the input sequence, in combination with the forward state, and then feedback the calculation results to the next neuron in the current recurrent layer and the corresponding neurons in the recurrent layer F2. Repeat the above steps for all neurons. After two layers of forward processing, the final state is obtained. ; The structures of the reverse recurrent layers B1 and B2 are the same as those of the recurrent layer. The difference is that the states of the two layers of the reverse recurrent layer are read from the end of the sequence and propagated from back to front to obtain the final output ; Step 3-3: An additional concatenation function is added between the B1 / F1 layer and the B2 / F2 layer in the interactive bidirectional RNN to concatenate the outputs of the recurrent layer F1 and the reverse recurrent layer B1 and feed them to the recurrent layer F2 and the reverse recurrent layer B2; both the recurrent layer F2 and the reverse recurrent layer B2 receive the same input and have an explicit connection; the calculation formula is as follows: Among them is the concatenation of forward states of the same size and backward states ; represents the GRU output of the F1 layer, represents the GRU output of the B1 layer, represents the GRU output of the F2 layer, represents the GRU output of the B2 layer; Step 4: Deep sparse penalty; During the training process, the optimization objective is as follows: Among them is the global error loss, defined as the cross-entropy between the model output and the true label; represents the sparsity penalty of the output matrix of the fully connected layer, using norm representation, used to adjust the weight of the sparsity penalty term; is the sparsity penalty of the weight matrix of the fully connected layer, as the weight parameter of this penalty term to control the sparsity; Step 5: After the training set and the test set are processed by Step 2, train and test the EEG signal classification model constructed in Step 3 to obtain the final EEG signal classification model McRFS-IBiRNN based on random recombination of multi-segment signals and interactive bidirectional RNN. The output of McRFS-IBiRNN is the final classification result.

2. The EEG signal classification method based on multi-segment signal random recombination and interactive bidirectional RNN according to claim 1, wherein The EEG signal is the motor / imagery EEG dataset provided by PhysioNet.

Images