Electroencephalogram signal classification method based on hybrid neural network

Abstract

The application discloses a kind of electroencephalogram classification method based on hybrid neural network, belong to man-machine interaction intelligent technical field.The purpose of the present application is to convert electroencephalogram into digital signal and form control instruction for controlling vehicle after accurate classification, realize electroencephalogram for automobile control auxiliary driving based on hybrid neural network electroencephalogram classification method.The steps of the present application are: collecting electroencephalogram, electroencephalogram data preprocessing, constructing fusion entropy matrix, training hybrid neural network model, data classification of test set, connecting host computer with simulated cockpit, driver wears electroencephalogram cap, test driver's control effect on vehicle.The present application greatly improves the accuracy of classification, and the model of hybrid neural network is optimized to obtain better classification effect.At the same time, the classification result of neural network is converted into control instruction for car, which can realize electroencephalogram control for vehicle in emergency, realize five kinds of control states of parking, left turn, right turn, acceleration and normal driving, which provides protection for the safety of driver and fellow passengers.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction intelligent technology. Background Technology

[0002] Brain-computer interface (BCI) technology is an interdisciplinary field that integrates psychology, neuroscience, cognitive science, biomedicine, and mathematics. The development of each discipline plays a crucial role in the advancement of BCI technology. BCI technology is a scientific technology that establishes control command transmission between the human brain and electronic devices such as computers without the need for peripheral nerves or limb involvement. In the past decade, BCI technology has made significant progress and rapid development, and its application areas are expanding. The basic principle of BCI technology is to collect electroencephalogram (EEG) signals from the human cerebral cortex using signal acquisition devices. These signals are then amplified, filtered, and noise-reduced. Feature extraction is performed on the processed signals, and these features are used to classify the EEG signals. Finally, the classification results are converted into control commands to control external devices.

[0003] Therefore, if BCI technology can be used to conduct research on intelligent assisted driving and decode the driver's driving intentions to control the vehicle, it will not only help improve and develop intelligent assisted driving systems, but also enhance driving safety. Summary of the Invention

[0004] The purpose of this invention is to convert EEG signals into digital signals after accurate classification and form control commands for controlling vehicles, thereby realizing a hybrid neural network-based EEG signal classification method for vehicle control and assisted driving.

[0005] The steps are:

[0006] S1. Construct the fusion entropy matrix:

[0007] The preprocessed data was divided into 500ms signal segments, and the marginal spectral entropy and fuzzy entropy were calculated for each segment:

[0008] S11, The marginal spectral entropy is as follows:

[0009] Applying a Hilbert transform to the IMF components yields the following equation:

[0010] (1)

[0011] Where PV is the Cauchy principal value;

[0012] get The analytical expression is

[0013] (2)

[0014] In the above formula Represents instantaneous amplitude. Represents instantaneous phase;

[0015] The instantaneous frequency of the Hilbert transform is defined as:

[0016] (3)

[0017] In summary, the signal before decomposition can be represented in the form of the sum of the transformed IMFs as follows:

[0018] (4)

[0019] Therefore, the Hilbert spectrum is defined as:

[0020] (5)

[0021] Hilbert's marginal spectral entropy is as follows:

[0022] (6)

[0023] S12, the fuzzy entropy is as follows:

[0024] For a given 3D time series: Define the dimension of phase space and the sequence Perform phase space reconstruction:

[0025] ;

[0026] in, The formula for the mean is as follows:

[0027] ;

[0028] definition For vectors and The distance between them is also the maximum difference between corresponding elements of the two vectors;

[0029] (9);

[0030] in ,and ;

[0031] Introducing fuzzy membership functions:

[0032] ;

[0033] Two vectors are obtained using fuzzy membership functions. and The similarity between them is as follows:

[0034] (11)

[0035] in It is an exponential function. and These represent the width and gradient of the exponential function, respectively.

[0036] For all Taking the average yields:

[0037] (12)

[0038] Let the dimension of the phase space be set to Repeat the above steps to obtain ;

[0039] Finally, the sequence length is The fuzzy entropy of the time series is as follows:

[0040] (13)

[0041] Typical time series Since the sequence is finite, the fuzzy entropy estimate is:

[0042] (14)

[0043] The value of fuzzy entropy is mainly related to related, Let the phase space dimension be , Similarity tolerance;

[0044] The marginal spectral entropy and fuzzy entropy are fused to construct a fused entropy matrix;

[0045] S2. Training the hybrid neural network model:

[0046] L0: This is the input layer. The input to this layer is the training set of the fusion entropy matrix constructed in step S1; the input fusion entropy matrix is ​​represented as... ,in The number of leads in the EEG acquisition device. Represented as the number of frequency points; using Indicates the first The first in the layer Each image The output value, then used To indicate the first The first in the layer Each image Input values:

[0047] (15)

[0048] in The activation function has the following expression:

[0049] (16)

[0050] In the expression These are hyperparameters, whose optimal values ​​are obtained through model training.

[0051] L1: (17)

[0052] (18)

[0053] (19)

[0054] in, This is the input matrix for the L1 layer. The size of the convolution kernel. Representing the The nth convolutional kernel The connection weights between each neuron and the corresponding position in the input matrix. For the bias of layer L1, This is the output of the matrix after passing through the activation function. for The final output of the L1 layer after group normalization;

[0055] L2: (20)

[0056] This layer is the second part of the hybrid neural network, consisting of a convolutional layer and a pooling layer. This is the input matrix for the L2 layer. This refers to the size of the convolutional kernel in this layer. For the t-th convolutional kernel in L2 layer Location and The connection weights at corresponding positions, For the bias of the L2 layer;

[0057] L3: (twenty one)

[0058] This layer consists of a single convolutional layer, in which This is the input matrix for the L3 layer. The vertical size of the convolution kernel. The horizontal size of the convolution kernel. For the t-th convolutional kernel in L3 layer Location and The connection weights at corresponding positions, For the bias of layer L3;

[0059] L4: (twenty two)

[0060] This layer is the fourth part of the hybrid neural network, consisting of one convolutional layer and one pooling layer. The vertical size of the convolution kernel. The horizontal size of the convolution kernel. For the t-th convolutional kernel in L4 layer Location and The connection weights at corresponding positions, For the bias of layer L4;

[0061] L5: (twenty three)

[0062] This layer is the fifth part of the hybrid neural network, consisting of a convolutional layer and a pooling layer; The vertical size of the convolution kernel. The horizontal size of the convolution kernel. For the t-th convolutional kernel in L5 layer Location and The connection weights at corresponding positions, For the bias of layer L5;

[0063] L6: (twenty four)

[0064] (25)

[0065] (26)

[0066] (27)

[0067] (28)

[0068] (29)

[0069] (30)

[0070] (31)

[0071] (32)

[0072] (33)

[0073] (34)

[0074] This layer is a bidirectional LSTM layer in a hybrid neural network. and This is the activation function for this layer. This represents the cell state in a forward LSTM layer. This represents the cell state in the inverse LSTM layer. These are the biases for the input gate, forget gate, and output gate in a forward LSTM, respectively. These represent the biases for the input gate, forget gate, and output gate in a reverse LSTM. These are the weights in a forward LSTM; These are the weights in the inverse LSTM;

[0075] L7: (35)

[0076] This layer is a fully connected layer in a hybrid neural network. and These represent the vertical and horizontal dimensions of the output matrix of the bidirectional LSTM layer, respectively. For the corresponding The connection weights in the middle, This is the bias of the layer;

[0077] L8: (36)

[0078] This layer is the final classification layer, where for The conditional probability generated by the function represents the probability under given input. In this case, the final classification is Conditional probability of a category;

[0079] S3. Data classification for the test set:

[0080] The test set data is input into the final classification model in step S2. The test data is divided into five categories: parking, left turn, right turn, acceleration, and normal driving, and the classification accuracy of the neural network model is obtained.

[0081] This invention improves upon traditional EEG extraction methods, extracting more effective EEG signals. It fuses a convolutional neural network with a bidirectional LSTM, using a fusion entropy matrix derived from the fusion of marginal spectral entropy and fuzzy entropy of the EEG signals as input, significantly enhancing classification accuracy. Furthermore, optimizing the hybrid neural network model yields even better classification results. Simultaneously, the classification results are converted into control commands for the vehicle, enabling EEG-based vehicle control in emergency situations, achieving five control states: parking, left turn, right turn, acceleration, and normal driving. This provides a safety guarantee for the driver and passengers. Attached Figure Description

[0082] Figure 1This is a flowchart of the overall experimental process of the present invention;

[0083] Figure 2 This is a flowchart of the method for training the hybrid neural network model of the present invention;

[0084] Figure 3 This is a structural diagram of the hybrid neural network model of the present invention. Detailed Implementation

[0085] This invention involves a user wearing an EEG acquisition device. After extracting the user's EEG signals, the device performs preprocessing, feature extraction, and action classification. The optimal classification method is determined by comparing multiple classifiers based on the user's EEG signals. After accurate classification, the EEG signals are converted into digital signals and used to generate control commands for vehicle control, enabling assisted driving using EEG signals. In summary, the EEG signal classification method based on a hybrid neural network for assisted driving described in this invention has certain rationality and advantages.

[0086] The problem this invention aims to solve is to avoid traffic accidents caused by the brain reacting before the hands and feet complete the corresponding actions in emergency situations, and to provide a brainwave signal classification method based on a hybrid neural network for assisted driving, facilitating roadside parking and passage through narrow roads. This method makes vehicle movement control more flexible and rapid, and it requires minimal operator skill; simply wearing an EEG acquisition device is sufficient to control the vehicle in emergency situations.

[0087] This invention includes the following steps:

[0088] Step 1: Collecting EEG signals: Subjects wear EEG acquisition devices and are placed in a simulated driving cockpit to extract EEG signals during the simulated driving process;

[0089] Step 2: EEG data preprocessing;

[0090] Step 3: Constructing the Fusion Entropy Matrix: To improve the accuracy of EEG signal classification, feature extraction is performed on the preprocessed EEG signals. This invention extracts the marginal spectral entropy from the time-frequency domain features of the EEG signals and the fuzzy entropy from the nonlinear features for subsequent classification. Fuzzy entropy adds a fuzzy factor to the sample entropy, exhibiting good continuity and robustness, and has a good effect on feature extraction of nonlinear complex EEG signals. The extracted marginal spectral entropy and fuzzy entropy are fused to construct a two-dimensional fusion entropy matrix. Its dimensions represent the number of leads of the EEG acquisition device and the number of frequency points after calculating the fuzzy entropy and marginal spectral entropy, respectively. The numerical values ​​represent the fuzzy entropy value and marginal spectral entropy value of a certain lead at a certain frequency point.

[0091] Step 4: Train the hybrid neural network model: Divide the extracted fusion entropy matrix into training and test sets in a 7:3 ratio. Construct a hybrid neural network model to classify the data in the training set, and use the Adam algorithm to optimize the weights and bias parameters in the neural network. Then, input the validation set data into the neural network model, and further adjust the hyperparameters of the neural network model based on the classification results of the validation set to obtain the best classification effect.

[0092] Step 5: Data classification of the test set: Input the test set data into the final classification model in Step 4. The test data is divided into five categories: parking, left turn, right turn, acceleration, and normal driving. The classification accuracy of the neural network model is obtained.

[0093] Step 6: Connect the host computer to the driver simulator. The driver wears an EEG cap to test the driver's control over the vehicle.

[0094] Step 1 involves the following steps: The subject wears an EEG acquisition device and is placed in a simulated driving cockpit. A large screen in the cockpit simulates real road conditions, and the subject drives according to the route displayed on the ground. Software programming is used to simulate the sudden appearance of vehicles and pedestrians, recording the subject's EEG signals during these emergency situations. Simultaneously, sudden left and right turn prompts are also set, and the subject's EEG signals are recorded during these emergency left and right turns. To increase the realism of the road simulation and the validity of the experimental data, the interval between two sudden situations should exceed ten minutes. Multiple trials are conducted on the same subject to obtain a large amount of EEG data.

[0095] Step 2 specifically involves filtering the EEG data to reduce noise between different channels. Artifacts such as scalp electromyography (EMG), electrooculography (EOG), and electrocardiogram (ECG) are removed from the EEG signals.

[0096] Step 3 specifically involves:

[0097] The preprocessed data was divided into 500ms signal segments, and the marginal spectral entropy and fuzzy entropy were calculated for each segment.

[0098] The specific calculation method for marginal spectral entropy is as follows:

[0099] The EEG signal is processed using an integrated empirical mode decomposition algorithm to obtain the IMF component. Applying a Hilbert transform to the obtained IMF component yields the following equation:

[0100] (1)

[0101] PV is the Cauchy principal value.

[0102] Then obtained The analytical expression is

[0103] (2)

[0104] In the above formula Represents instantaneous amplitude. Represents the instantaneous phase.

[0105] The instantaneous frequency of the Hilbert transform is defined as:

[0106] (3).

[0107] In summary, the signal before decomposition can be represented as the sum of the transformed IMF values ​​as follows:

[0108] (4).

[0109] Therefore, the Hilbert spectrum is defined as:

[0110] (5).

[0111] Hilbert's marginal spectral entropy (MS) is as follows:

[0112] (6).

[0113] The specific calculation method for fuzzy entropy is as follows:

[0114] For a given N-dimensional time series: Define the dimension of phase space And perform phase space reconstruction on sequence U:

[0115] (7)

[0116] in, The mean is calculated using the following formula:

[0117] (8)

[0118] definition For vectors and The distance between them is also the maximum difference between corresponding elements of the two vectors.

[0119] The expression is:

[0120] (9)

[0121] in, and .

[0122] Introducing fuzzy membership functions:

[0123] (10).

[0124] Two vectors are obtained using fuzzy membership functions. and The similarity between them is as follows:

[0125] (11)

[0126] Among them It is an exponential function. and These represent the width and gradient of the exponential function, respectively.

[0127] For all Taking the average, we get:

[0128] (12)

[0129] Let the dimension of the phase space be set to Repeat the above steps to obtain .

[0130] Finally, the fuzzy entropy of a time series with a sequence length of N is as follows:

[0131] (13).

[0132] Since the time series N is usually finite, the fuzzy entropy can be approximately estimated as follows:

[0133] (14)

[0134] The value of fuzzy entropy is mainly related to related, Let the phase space dimension be , This is a similarity tolerance.

[0135] This application sets , ,in denoted as the standard deviation of the time series.

[0136] The calculated marginal spectral entropy and fuzzy entropy are fused to construct a fused entropy matrix with 64 leads and 470 frequency points.

[0137] Step 4 specifically involves:

[0138] The neural network model consists of a convolutional neural network and a bidirectional LSTM connection. It comprises an input layer, a first convolutional layer, a second convolutional layer, a first pooling layer, a third convolutional layer, a fourth convolutional layer, a second pooling layer, a fifth convolutional layer, a third pooling layer, a bidirectional LSTM layer, a fully connected layer, and a final output layer. The model's input is the fuzzy entropy matrix constructed in step 3. The convolutional layers are used to actively learn and extract translation-invariant feature maps of the input signal, while the pooling layers are used for dimensionality reduction. In the convolutional neural network portion, the ELU function is used as the activation function. To avoid the constraint of the number of samples in the batch during normalization, group normalization (GN) is used to normalize the data. Random deactivation is employed to prevent overfitting during model training. Cross-validation showed that a dropout rate of 0.5 yielded the best results because a dropout rate of 0.5 generates the most randomly generated network structures. The final output layer uses the softmax function, whose output is the conditional probability of classifying a given input as a certain label. The parameters in the model are adjusted by minimizing the loss value for each classification, using the cross-entropy loss function. The Adam algorithm is then used to optimize the parameters in the neural network model until the model achieves optimal classification performance.

[0139] The specific components of a neural network model are:

[0140] L0: This is the input layer. The input to this layer is the training set of the fusion entropy matrix constructed in step 3. The input fusion entropy matrix is ​​represented as follows: ,in The number of leads in the EEG acquisition device. This is represented as the number of frequency points. The input matrix size is 64*470. (Using...) Indicates the first The first in the layer Each image The output value, then used To indicate the first The first in the layer Each image Input values:

[0141] (15)

[0142] in This is the activation function.

[0143] The activation function used in this invention is the ELU function, and the expression of the ELU function is as follows:

[0144] (16)

[0145] In the expression These are hyperparameters, and their optimal values ​​can be obtained through model training. The activation function works on the data while also performing group normalization (GN) on the data.

[0146] L1: (17)

[0147] (18)

[0148] (19)

[0149] in, This is the input matrix for the L1 layer. is the size of the convolution kernel, which is 3. Representing the The nth convolutional kernel The connection weights between each neuron and the corresponding position in the input matrix. This is the bias for layer L1. This is the output of the matrix after passing through the activation function. for The final output of the L1 layer after group normalization. Specifically, in this invention, this layer extracts the feature map of the input matrix using 10 convolutional kernels of size 1*3, and obtains the output value of each neuron in the L1 layer under the action of the activation function ELU function. After convolution operation, the output dimension of the L1 layer is 64*468*10.

[0150] L2: (20)

[0151] This layer is the second part of the hybrid neural network, consisting of a convolutional layer and a pooling layer. This is the input matrix for the L2 layer. This represents the size of the convolutional kernel for this layer, which is 64. For the t-th convolutional kernel in L2 layer Location and The connection weights at corresponding positions, This is the bias for the L2 layer. Specifically, this layer uses a 1*64 convolutional kernel to eliminate the non-spatial arrangement of lead dimensions in the fuzzy entropy matrix, resulting in an output dimension of 10*468. The output of the convolutional layer is then fed into a pooling layer, which uses a 1*4 average pooling method. The final output dimension of the L2 layer is 10*117.

[0152] L3: (twenty one)

[0153] This layer consists of a single convolutional layer, in which This is the input matrix for the L3 layer. is the vertical size of the convolution kernel, and its value is 10; is the horizontal size of the convolution kernel, and its value is 5. For the t-th convolutional kernel in L3 layer Location and The connection weights at corresponding positions, This is the bias of the L3 layer. Specifically, this layer uses 20 convolutional kernels of size 10*5 to extract the translation-invariant feature map of the L2 layer output matrix, and obtains the output value of each neuron in the L3 layer under the action of the ELU activation function. Finally, the output dimension of L3 is 20*113.

[0154] L4: (twenty two)

[0155] This layer is the fourth part of the hybrid neural network, consisting of a convolutional layer and a pooling layer. is the vertical size of the convolution kernel, and its value is 10; is the horizontal size of the convolution kernel, and its value is 6. For the t-th convolutional kernel in L4 layer Location and The connection weights at corresponding positions, This is the bias for layer L4. Specifically, this layer uses 20 convolutional kernels of size 10*6 to extract the translation-invariant feature map of the output matrix of layer L3. Under the action of the ELU activation function, the output values ​​of each neuron in layer L4 are obtained, resulting in an output dimension of 20*108 for the convolutional layer in L4. The output of the convolutional layer is then fed into a pooling layer, which uses a 1*4 average pooling method. The final output dimension of layer L4 is 20*27.

[0156] L5: (twenty three)

[0157] This layer is the fifth part of the hybrid neural network, consisting of a convolutional layer and a pooling layer. The vertical size of the convolution kernel is 20; is the horizontal size of the convolution kernel, and its value is 4. For the t-th convolutional kernel in L5 layer Location and The connection weights at the corresponding positions are the biases of the L5 layer. Specifically, this layer uses 30 convolutional kernels of size 20*4 to extract the translation-invariant feature map of the L4 layer output matrix. Under the action of the ELU activation function, the output values ​​of each neuron in the L5 layer are obtained, resulting in an output dimension of 30*24 for the convolutional layer in L5. The output of the convolutional layer is then fed into a pooling layer, which uses a 1*3 average pooling method. The final output dimension of the L4 layer is 30*8.

[0158] L6: (twenty four)

[0159] (25)

[0160] (26)

[0161] (27)

[0162] (28)

[0163] (29)

[0164] (30)

[0165] (31)

[0166] (32)

[0167] (33)

[0168] (34)

[0169] This layer is a bidirectional LSTM layer in a hybrid neural network. Because EEG signals have temporal characteristics, a recurrent neural network is added after the convolutional neural network. Furthermore, since the current state of the EEG signal is related to the EEG signals generated before and after it, a bidirectional LSTM is used to process the data. and The activation function for this layer is selected in this invention. For the sigmoid function, This refers to the tanh function. This represents the cell state in a forward LSTM layer. This represents the cell state in the inverse LSTM layer. These are the biases for the input gate, forget gate, and output gate in a forward LSTM, respectively. These are the biases for the input gate, forget gate, and output gate in an inverse LSTM, respectively. These are the weights in a forward LSTM; These are the weights in the inverse LSTM. The optimal values ​​are determined through continuous training. The bidirectional LSTM contains 30 neurons in both the forward and backward directions. The 30*8 matrix output from the convolutional neural network is input into each of these 30 neurons according to its dimension. After processing by the bidirectional LSTM, a 30*40 dimension output matrix is ​​finally obtained.

[0170] L7: (35)

[0171] This layer is a fully connected layer in a hybrid neural network. m and n are the vertical and horizontal dimensions of the output matrix of the bidirectional LSTM layer, respectively. The value of m ranges from 1 to 30, and the value of n ranges from 1 to 40. For the corresponding The connection weights in the middle, This is the bias of the layer. The layer first uses the `flatten` function to flatten the output matrix of layer L6 into a one-dimensional output of size 1*1200. Then, the flattened matrix is ​​processed, resulting in a final output matrix of size 1*400.

[0172] L8: (36)

[0173] This layer is the final classification layer, where for The conditional probability generated by the function represents the probability under given input. In this case, the final classification is Conditional probabilities of categories. The classification in this invention includes five categories: parking, left turn, right turn, acceleration, and normal driving, labeled as 1, 2, 3, 4, and 5, respectively. The result after function processing is a probability distribution. By comparing the result obtained from the hybrid neural network model with the actual category of the input EEG signal, the Adam algorithm is used to optimize the parameters of each component in the hybrid neural network model until the classification model achieves the optimal classification effect.

[0174] for The layer with the highest probability of correctly classifying a category has the most complex classification process. This invention uses the cross-entropy loss function to calculate the loss; the smaller the cross-entropy, the more accurate the classification result. The cross-entropy is used to continuously optimize the entire model to achieve the optimal classification effect. To prevent overfitting during model training, a random dropout method with a dropout rate of 0.5 is used.

[0175] Step 5 specifically involves:

[0176] The test set divided in step 4 is put into the hybrid neural network obtained in step 4 for testing. The model is classified on the test set using 10-fold cross-validation and the final classification effect is observed.

[0177] Step 6 specifically involves:

[0178] A host computer equipped with a hybrid neural network is connected to a driving simulator, and the classification results of the neural network are converted into control commands and input into the simulator. Then, a driver who has had their EEG signals collected wears an EEG acquisition device to simulate driving. The driver's EEG signals are extracted in real time and fuzzy entropy is calculated and classified using the hybrid neural network. The program simulates emergency situations encountered in real driving, such as suddenly appearing pedestrians, vehicles, and objects. After the EEG signals are recognized, they not only control the simulated car but also display the generated commands in text form on the host computer for easy observation of the classification effect of the EEG signals in emergency situations.

[0179] The present invention will now be described in detail with reference to the accompanying drawings:

[0180] Figure 1 The overall flowchart of this invention is as follows: The subject wears an EEG acquisition device and drives in a simulated cockpit according to a route displayed on the ground on a screen. Software programming is used to simulate the sudden appearance of vehicles and pedestrians, recording the subject's EEG signals in emergency situations. Simultaneously, sudden left and right turn prompts are set, and the subject's EEG signals are recorded during emergency left and right turns. The acquired EEG signals are processed using the common average reference method and independent component analysis to remove noise and artifacts. The preprocessed EEG signals are then used to calculate marginal spectral entropy and fuzzy entropy, which are fused to form a fusion entropy matrix. The training set defined by the fusion entropy matrix is ​​fed into a hybrid neural network to train the model. The test set defined by the fusion entropy matrix is ​​then fed into the trained hybrid neural network model for classification, and the classification accuracy is observed. Finally, the classification algorithm is connected to the simulated cockpit to observe the classification effect in practical applications.

[0181] Figure 2 The training flowchart for the hybrid neural network model in this invention is as follows:

[0182] Step 1: Subjects wear an EEG acquisition device and are placed in a cockpit to simulate driving, completing actions according to different emergency situations and instructions.

[0183] The second step involves collecting the driver's EEG signals and processing them using the common average reference method and independent component analysis to remove noise and artifacts.

[0184] Step 3: Divide the preprocessed data into 500ms signal segments, and calculate the marginal spectral entropy and fuzzy entropy for each segment.

[0185] For detailed calculation methods of marginal spectral entropy and fuzzy entropy, please refer to sections

[0014] -

[0028] .

[0186] Step 4: Construction and training of the hybrid neural network model. Figure 3 The specific structure of the hybrid neural network model constructed in this invention is as follows:

[0187] L0: Input layer. The input to this layer is the training set of the constructed fusion entropy matrix. The size of the input matrix is ​​64*470, where 64 represents the number of leads in the EEG acquisition device and 470 represents the number of frequency points.

[0188] L1: This layer is the first part of the hybrid neural network, consisting of a single convolutional layer with 10 1x3 kernels for data processing. The ELU function is used as the activation function to activate the input matrix. The parameters of the activation function are determined through training, and the data is group normalized. The final output matrix has dimensions of 64*468*10.

[0189] L2: This layer is the second part of the hybrid neural network, consisting of a convolutional layer and a pooling layer. To eliminate the non-spatial nature of the lead dimension arrangement in the fuzzy entropy matrix, the convolutional layer uses a 1*64 convolutional kernel for data processing. The ELU function is used as the activation function to activate the input matrix. The parameters of the activation function are determined through training, and group normalization is performed on the data. The resulting 10*468 matrix is ​​then processed by a 1*4 average pooling method, resulting in a final output matrix with a dimension of 10*117.

[0190] L3: This layer is the third part of the hybrid neural network, consisting of a single convolutional layer with 20 10*5 kernels for data processing. The ELU function is used as the activation function to activate the input matrix. The parameters of the activation function are determined through training, and the data is group normalized. The final output matrix has a dimension of 20*113.

[0191] L4: This layer is the fourth part of the hybrid neural network, consisting of one convolutional layer and one pooling layer. The convolutional layer uses 20 convolutional kernels of size 10*6 for data processing. The ELU function is used as the activation function to activate the input matrix. The parameters of the activation function are determined through training, and group normalization is performed on the data. The resulting 20*108 matrix is ​​then subjected to 1*4 average pooling, resulting in a final output matrix with dimensions of 20*27.

[0192] L5: This layer is the fifth part of the hybrid neural network, consisting of one convolutional layer and one pooling layer. The convolutional layer uses 30 convolutional kernels of size 20*4 for data processing. The ELU function is used as the activation function to activate the input matrix. The parameters of the activation function are determined through training, and group normalization is performed on the data. The resulting 30*24 matrix is ​​then subjected to 1*3 average pooling, resulting in a final output matrix with a dimension of 30*8.

[0193] L6: This layer is the sixth part of the hybrid neural network, consisting of a single bidirectional LSTM layer. Because EEG signals have temporal characteristics, a recurrent neural network is added after the convolutional neural network. Furthermore, since the current state of the EEG signal is related to the EEG signals generated before and after it, a bidirectional LSTM is used to process the data. The bidirectional LSTM contains 30 neurons in both the forward and backward directions. The 30*8 matrix output from the convolutional neural network is input into each of the 30 neurons according to its dimension. After processing by the bidirectional LSTM, a 30*40 dimension output matrix is ​​finally obtained.

[0194] L7: This layer is the seventh part of the hybrid neural network, consisting of a fully connected layer. This layer first uses the flatten function to flatten the output matrix of layer L6 into a one-dimensional output of size 1*1200. Then, the flattened matrix is ​​processed, resulting in a final output matrix of size 1*400.

[0195] L8: This layer is the eighth part of the hybrid neural network and is the final classification layer. The classification in this invention includes five categories: parking, left turn, right turn, acceleration, and normal driving, labeled 1, 2, 3, 4, and 5 respectively. The result after processing by the Softmax function is a probability distribution. By comparing the results obtained from the hybrid neural network model with the actual category of the input EEG signal, the Adam algorithm is used to optimize the parameters of each component in the hybrid neural network model until the classification model reaches its optimal performance. To maximize the probability of correctly classifying the target category during Softmax classification, this invention uses the cross-entropy loss function to calculate the loss; the smaller the cross-entropy, the more accurate the classification result. Cross-entropy is used to continuously optimize the entire model to achieve the optimal classification effect. To prevent overfitting during model training, a dropout rate of 0.5 is used for random deactivation.

[0196] Methods used in designing hybrid neural networks:

[0197]

[0198] Step 5: Put the test set divided in the fusion entropy matrix into the optimal hybrid neural network obtained by final training for testing. Use 10 times 10-fold cross-validation to classify the test set and observe the final classification effect.

[0199] Step 6: Connect the host computer with the hybrid neural network to the driving simulator, and convert the classification results of the neural network into control commands, which are then input into the driving simulator. Next, have the driver, whose EEG signals have been collected, wear an EEG acquisition device to simulate driving. The driver's EEG signals are extracted in real time, and marginal spectral entropy and fuzzy entropy are calculated and classified using the hybrid neural network. The program is used to simulate emergency situations encountered in real driving, such as suddenly appearing pedestrians, vehicles, or objects. After the EEG signals are recognized, they not only control the simulated car but also display the generated commands in text form on the host computer for easy observation of the EEG signal classification effect under emergency conditions.

[0200] This invention improves upon traditional EEG extraction methods, extracting more effective EEG signals. It fuses a convolutional neural network with a bidirectional LSTM, selecting the marginal spectral entropy and fuzzy entropy of the EEG signals and using the resulting fusion entropy matrix as input, significantly improving classification accuracy. Furthermore, optimizing the hybrid neural network model yields even better classification results. Simultaneously, the classification results are converted into control commands for the vehicle, enabling EEG-based vehicle control in emergency situations, achieving five control states: parking, left turn, right turn, acceleration, and normal driving. This provides a safety guarantee for the driver and passengers.

Claims

1.A method for electroencephalogram signal classification based on hybrid neural networks, characterized in that: The steps are: S1, constructing a fusion entropy matrix: The preprocessed data is segmented into 500ms signal segments, and the marginal spectrum entropy and fuzzy entropy of each segment of data are calculated: S11, the marginal spectrum entropy is as follows: The Hilbert transform of the IMF component is obtained as follows: （1） Where PV is the Cauchy principal value; The analytical expression for is （2） In the above formulae represents the instantaneous amplitude, represents the instantaneous phase; The instantaneous frequency of the Hilbert transform is defined as: （3） In summary, the signal before decomposition is expressed as the sum of the transformed IMF, and the form after transformation is: （4） Then, the Hilbert spectrum is defined as: （5） The Hilbert marginal spectrum entropy is as follows: （6） S12, the fuzzy entropy is as follows: For a given dimensional time series: , define the phase space dimension and reconstruct the phase space for the series : ； wherein is the mean value, which is calculated as follows: ； Definitions distance between vectors and is the maximum difference between corresponding elements of the two vectors; （9） wherein , and ; A fuzzy membership function is introduced: ； The similarity between two vectors is obtained by means of fuzzy membership functions as follows: and ​ （11） wherein is an exponential function, and denote the width and gradient of the exponential function, respectively; For all Average: （12） The phase space dimension is set to , the above steps are repeated to obtain ; Finally, the fuzzy entropy of the time series of length is as follows: （13） The sequence of the usual time series is finite, so the fuzzy entropy is estimated as: （14） The value of fuzzy entropy is mainly related to , the phase space dimension, the similarity tolerance; The marginal spectrum entropy and fuzzy entropy are fused to construct a fusion entropy matrix; S2, training a hybrid neural network model: L0: This is the input layer. The input to this layer is the training set of the fusion entropy matrix constructed in step S1; the input fusion entropy matrix is ​​represented as... ,in The number of leads in the EEG acquisition device. Represented as the number of frequency points; using Indicates the first The first in the layer Each image The output value, then used To indicate the first The first in the layer Each image Input values: （15） wherein is an activation function, whose expression is as follows: （16） in the expression are hyperparameters, whose optimal values are obtained from model training; L1： （17） （18） （19） in, This is the input matrix for the L1 layer. The size of the convolution kernel. Representing the The nth convolutional kernel The connection weights between each neuron and the corresponding position in the input matrix. For the bias of layer L1, This is the output of the matrix after passing through the activation function. for The final output of the L1 layer after group normalization; L2： （20） The layer is the second part of the mixed neural network, which is composed of a convolution layer and a pooling layer, wherein is the input matrix of the L2 layer, is the size of the convolution kernel of the layer, is the position of the tth convolution kernel of the L2 layer, is the connection weight of the corresponding position, is the bias of the L2 layer; L3： （21） The layer is composed of one convolution layer, wherein is an input matrix of the L3 layer, is a longitudinal size of the convolution kernel, is a transverse size of the convolution kernel, is a position in the tth convolution kernel of the L3 layer, connection weight of the corresponding position, is a bias of the L3 layer; L4： （22） The layer is the fourth part of the mixed neural network, which is composed of a convolutional layer and a pooling layer, is the longitudinal size of the convolution kernel, is the transverse size of the convolution kernel, is the position of the connection weight of the corresponding position, is the bias of the L4 layer; L5： （23） The layer is the fifth part of the mixed neural network, which is composed of a convolution layer and a pooling layer; is the longitudinal size of the convolution kernel, is the transverse size of the convolution kernel, is the position of the tth convolution kernel in the L5 layer, is the connection weight of the position and the corresponding position, is the connection weight of the position and the corresponding position, is the bias of the L5 layer; L6： （24） （25） （26） （27） （28） （29） （30） （31） （32） （33） （34） The layer is a bidirectional LSTM layer in the hybrid neural network, and is an activation function for the layer, is a cell state in the forward LSTM layer; is a cell state in the backward LSTM layer, are biases for the input gate, the forget gate and the output gate in the forward LSTM, respectively; are biases for the input gate, the forget gate and the output gate in the backward LSTM, respectively, is a weight in the forward LSTM; is a weight in the backward LSTM; L7： （35） The layer is a full connection layer in the mixed neural network, and are respectively a longitudinal dimension and a transverse dimension of an output matrix of a bidirectional LSTM layer, is a connection weight in corresponding to is a bias of the layer; L8： （36） This layer is the final classification layer, where is the conditional probability generated by the function, which represents the conditional probability of the final classification being class, given the input x. S3, data classification of the test set: The data of the test set is input into the final classification model in step S2, and the test data is classified into five categories: parking, left turn, right turn, acceleration, and normal driving, to obtain the classification accuracy of the neural network model.

Images