An EEG Emotion Recognition Method Based on Spiking Neural Networks

Abstract

The present invention discloses an EEG emotion recognition method based on a spiking neural network. First, EEG data is collected and segmented in the time direction. Then, ANN and SNN models with the same network structure are constructed. The EEG data is input into the ANN model for training. When the accuracy of the ANN model trained by backpropagation is stable, the training of the ANN model is completed. Finally, the weights of the trained ANN model are used to initialize the weights of the SNN model, and the weights and internal parameters of the SNN are co-trained using a hybrid loss and backpropagation. When the accuracy of the model is stable, the training is ended, and a trained SNN model is obtained. The method of the present invention integrates biologically reasonable internal parameters into spiking neurons, thereby optimizing information processing at the neuron level and stabilizing signal transmission. Through the use of cross-model initialization and fusion loss optimization to achieve co-training of the SNN model, the SNN model can be used for EEG emotion recognition, significantly improving the accuracy and stability of EEG emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the technical fields of affective computing and brain-inspired intelligence applications, and particularly relates to an EEG emotion recognition method based on a spiking neural network. Background Art

[0002] Accurately detecting and decoding emotional states through physiological signals is a key challenge in the design and development of autonomous intelligent brain-computer interfaces (BCIs). Among various physiological signals, electroencephalogram (EEG) has become a promising direction for reliable emotion recognition due to its rich temporal dynamics and ability to capture real-time neural activities.

[0003] In recent years, some models based on spiking neural networks (SNNs) have been proposed for analyzing EEG signals. These models utilize the advantages of SNNs in processing temporal data, and their performance does not depend on complex architecture designs and information interactions. Although certain progress has been made, further improving the performance of SNN models still faces many challenges. Previous studies have successfully applied the introduction of gradient approximation functions in the backpropagation algorithm to the training of SNN models. This method effectively alleviates the non-differentiability problem of spike activities, thereby reducing the difficulty of model training. Another widely adopted strategy is to convert pre-trained artificial neural network (ANN) models into SNNs, avoiding the complexity of directly training SNN models. However, these methods usually ignore the rich intrinsic characteristics of biological neurons and fail to fully exploit the information and knowledge contained in pre-trained ANN models. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides an EEG emotion recognition method based on a spiking neural network, which uses cross-model initialization and fusion loss optimization to achieve collaborative training of the SNN model, significantly improving the accuracy and stability of EEG emotion recognition.

[0005] The technical solution adopted by the present invention is as follows: An EEG emotion recognition method based on a spiking neural network, and the specific steps are as follows:

[0006] S1. Collect EEG data and segment the EEG data in the time direction;

[0007] Collect an EEG signal X C×N , and segment it into T segments in the time direction. Then the expression of the EEG data is as follows:

[0008]

[0009] Among them, C represents the number of channels, and N represents the number of sampling points.

[0010] S2. Construct ANN and SNN models with the same network structure;

[0011] S21. Construct an ANN model;

[0012] The ANN model proposes a new spiking neural network structure for time series data processing, including: a fully connected structure FC and a one-dimensional convolutional structure Conv1d.

[0013] Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories. The size of the convolutional kernel in the one-dimensional convolutional structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories.

[0014] The ANN model is built using ReLU neurons or Sigmoid neurons, and the EEG data obtained in step S1 is used as the input data, then the output Y of the model ANN The expression is as follows:

[0015]

[0016] Among them, in the ANN model, the trainable parameter is the weight W of the model ANN .

[0017] S22. Use an improved LIF model to construct an SNN model;

[0018] Among them, the SNN model and the ANN model in step S21 have exactly the same network structure. The SNN model uses improved LIF spiking neurons, and co-learns the biological parameters of the improved LIF model, that is, integrates the biologically reasonable intrinsic parameters into the spiking neurons; then the iterative formula of the neuron membrane potential u t is as follows:

[0019] u t = α t ·u t-1 + I t ,

[0020] Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and spike firing situation of the neuron at time t, and α t represents the membrane potential decay coefficient. Then the expression of the neuron firing o t is as follows:

[0021] o t = H(u t - V th ),

[0022] Among them, V threpresents the neuron firing threshold, and H(·) represents the step activation function. When the neuron membrane potential is greater than the firing threshold, then o t = 1, otherwise o t = 0. And the membrane potential decay coefficient α t has the following expression:

[0023]

[0024] where τ represents the membrane time constant.

[0025] The input and output of the improved LIF model are pulse sequences, and the trainable parameters include the membrane time constant and the firing threshold. Using the improved LIF neurons to build an SNN model, the trainable parameters of the SNN model include: the weight W SNN of the model, the firing threshold V th and the membrane time constant τ. Using the EEG data obtained in step S1 as the input data, the output Y SNN of the SNN model has the following expression:

[0026]

[0027] where, O t represents the firing pulse of the SNN output layer at time t.

[0028] S3. Based on step S2, propose a supervised learning method for the ANN model based on time averaging. Use the backpropagation method and data labels to pre-train the ANN model. When the loss function of the ANN model remains unchanged, that is, after the accuracy is stable, stop the model training to obtain the trained ANN model;

[0029] The output of the ANN model is Y ANN in step S21, and the data label Lable EEG is the true category of the EEG data. Use the output of the ANN model and the data labels for supervised training, and the obtained loss function value L ANN has the following expression:

[0030] L ANN = MSE(Y ANN - Lable EEG ),

[0031] where MSE represents the mean squared error.

[0032] Finally, use the loss function value and the backpropagation method to train the parameters of the ANN model to obtain the trained ANN model, and the weight parameter of the trained ANN model is W ANN .

[0033] S4. Based on step S3, co-train the parameters of the SNN model to obtain a trained SNN model;

[0034] S41. Cross-model initialization;

[0035] Utilize the same structure between the ANN and SNN models, and use the weights W of the trained ANN model in step S3 ANN to initialize the weights of the SNN model, that is, directly transfer W ANN to the SNN model. At the same time, initialize the biological parameters of the SNN model, the membrane time constant τ and the firing threshold V th ;

[0036] S42. Hybrid loss optimization;

[0037] Use the prediction result Y of the trained ANN model ANN and the EEG data label Lable EEG to jointly construct a hybrid loss function, and use the hybrid loss function to supervise the training of the SNN model.

[0038] During the training process of the SNN model, the hybrid loss function Loss includes: information loss L info and label loss L label , and the expression is as follows:

[0039] Loss = L label + W info ·L info ,

[0040] where W info represents the weight controlling the information constraint, and L label represents the mean squared error MSE between the true label and the predicted value Y of the SNN model SNN . The information loss L info is calculated based on the KL divergence between the information label info_label generated from the ANN output and the BISNN predicted value Y SNN . The definition expression of the information label is as follows:

[0041] info_label = softmax(Y ANN / temp)

[0042] where temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model.

[0043] Based on the hybrid loss function value Loss, use the backpropagation method to co-train the weight parameter W SNN of the SNN model, the firing threshold V thThe membrane time constant τ, when the loss function of the SNN model remains unchanged, that is, after the accuracy is stable, stop the model training to obtain the trained SNN model.

[0044] Advantages of the present invention: The method of the present invention first collects EEG data and divides the EEG data in the time direction, then constructs ANN and SNN models with the same network structure, inputs the EEG data into the ANN model for training, and when the accuracy of the ANN model trained by backpropagation is stable, complete the training of the ANN model. Finally, use the weights of the trained ANN model to initialize the weights of the SNN model, and use the hybrid loss and backpropagation to co-train the weights and internal parameters of the SNN. When the accuracy of the model is stable, end the training to obtain the trained SNN model. The method of the present invention integrates biologically reasonable internal parameters (firing threshold and membrane time constant) into the spiking neurons, thereby optimizing information processing at the neuron level and stabilizing signal transmission. Through the use of cross-model initialization and fusion loss optimization to achieve co-training of the SNN model, the SNN model can be used for EEG emotion recognition, significantly improving the accuracy and stability of EEG emotion recognition. Description of the Drawings

[0045] Figure 1 It is a flowchart of an EEG emotion recognition method based on a spiking neural network of the present invention.

[0046] Figure 2 It is a schematic diagram of co-learning neurons in an embodiment of the present invention. Detailed Embodiments

[0047] The following further describes the method of the present invention in conjunction with the drawings and embodiments.

[0048] As Figure 1 shown, a flowchart of an EEG emotion recognition method based on a spiking neural network of the present invention is as follows:

[0049] S1. Collect EEG data and divide the EEG data in the time direction;

[0050] Collect a segment of EEG signal X C×N , and divide it into T segments in the time direction. Then the EEG data expression is as follows:

[0051]

[0052] Among them, C represents the number of channels, and N represents the number of sampling points.

[0053] S2. Construct ANN and SNN models with the same network structure;

[0054] S21. Construct an ANN model;

[0055] In this embodiment, the network structure is shown in Table 1. The ANN model proposes a new spiking neural network structure for processing time series data (time series signal classification), including: a fully connected structure (FC) and a one-dimensional convolutional structure (Conv1d).

[0056] Table 1

[0057]

[0058] Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories. The size of the convolutional kernel in the one-dimensional convolutional structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories.

[0059] The ANN model is built using ReLU neurons or Sigmoid neurons, and the EEG data obtained in step S1 is used as the input data, and the output Y ANN of the model is as follows:[[]]END]]

[0060]

[0061] Among them, in the ANN model, the trainable parameter is the weight W ANN .

[0062] S22. Build an SNN model using an improved LIF model;

[0063] Among them, the SNN model has exactly the same network structure as the ANN model in step S21. The SNN model uses an improved leaky-integrate and fire (LIF) spiking neuron. The SNN model is built using an improved LIF neuron, and it is proposed to co-learn the biological parameters of the improved LIF model, that is, to integrate the biologically reasonable intrinsic parameters (firing threshold and membrane time constant) into the spiking neuron. The schematic diagram of co-learning neurons in this embodiment is as Figure 2 shown; then the iterative formula of the neuron membrane potential u t is as follows:

[0064] u t = α t ·u t-1 + I t ,

[0065] Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and spike firing situation of the neuron at time t, and αt Represents the membrane potential decay coefficient. Then the neuron fires o t The expression is as follows:

[0066] o t = H(u t - V th ),

[0067] where V th represents the neuron firing threshold, and H(·) represents the step activation function. When the neuron membrane potential is greater than the firing threshold, then o t = 1, otherwise o t = 0. And the membrane potential decay coefficient α t The expression is as follows:

[0068]

[0069] where τ represents the membrane time constant.

[0070] The input and output of the improved LIF model are pulse sequences, and the trainable parameters include the membrane time constant and the firing threshold. Using the improved LIF neurons to build an SNN model, the trainable parameters of the SNN model include: the weight W of the model SNN , the firing threshold V th and the membrane time constant τ. Using the EEG data obtained in step S1 as the input data, the output Y of the SNN model SNN The expression is as follows:

[0071]

[0072] where O t represents the firing pulse of the SNN output layer at time t.

[0073] S3. Based on step S2, propose a supervised learning method for the ANN model based on time averaging. Use the backpropagation method and data labels to pre-train the ANN model. When the loss function of the ANN model remains unchanged, that is, when the accuracy is stable, stop the model training to obtain the trained ANN model;

[0074] The output of the ANN model is Y in step S21 ANN , and the data label Lable EEG is the true category of the EEG data (in a three-classification task in this embodiment, the categories are defined as 100, 010, and 001). Use the output of the ANN model and the data labels for supervised training, and the obtained loss function value L ANN The expression is as follows:

[0075] L ANN = MSE(YANN -Lable EEG ),

[0076] Among them, MSE represents the minimum mean square error.

[0077] Finally, the parameters of the ANN model are trained using the loss function value and the backpropagation method to obtain a trained ANN model, and the weight parameters of the trained ANN model are W ANN .

[0078] S4. Based on step S3, co-train the parameters of the SNN model to obtain a trained SNN model;

[0079] S41. Cross-model initialization;

[0080] Using the same structure between the ANN and SNN models, use the weight W of the ANN model trained in step S3 ANN to initialize the weights of the SNN model, that is, transfer W ANN directly into the SNN model. At the same time, initialize the biological parameters of the SNN model, the membrane time constant τ and the firing threshold V th In this embodiment, the membrane time constant τ and the firing threshold V th are respectively set to 2.0 and 1.0 mV;

[0081] S42. Hybrid loss optimization;

[0082] Use the prediction result Y of the trained ANN model ANN and the EEG data label Lable EEG to construct a hybrid loss function together, and use the hybrid loss function to supervise and train the SNN model.

[0083] During the training process of the SNN model, the hybrid loss function Loss includes: information loss (L info ) and label loss (L label ), and the expression is as follows:

[0084] Loss = L label + W info · L info ,

[0085] Among them, W info represents the weight controlling the information constraint, and L label represents the mean square error (MSE) between the true label and the predicted value Y SNN of the SNN model. The information loss L info is based on the mean square error between the information label (info_label) generated by the ANN output and the BISNN predicted value Y SNNCalculated by the Kullback-Leibler (KL) divergence between them. The information label is defined by the following expression:

[0086] info_label = softmax(Y ANN / temp)

[0087] where temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model.

[0088] Based on the value of the hybrid loss function Loss, using the backpropagation method, co-train the weight parameters W of the SNN model SNN , the firing threshold V th and the membrane time constant τ. When the loss function of the SNN model remains unchanged, that is, when the accuracy is stable, stop the model training to obtain the trained SNN model.

[0089] This embodiment further conducts simulation verification. Use the two network structures provided in Table 1 to construct the SNN model in the method of the present invention, test on the data of Arousal and Valence, the hyperparameters are set as shown in Table 2, and the experimental results are shown in Table 3.

[0090] Table 2

[0091]

[0092] Table 3

[0093]

[0094] As can be seen from Tables 2 and 3, compared with the existing SNN models, the SNN model of the method of the present invention has higher emotion recognition accuracy and more stable results (i.e., lower variance).

[0095] In summary, the method of the present invention integrates biologically reasonable internal parameters (firing threshold and membrane time constant) into spiking neurons, thereby optimizing information processing at the neuron level and stabilizing signal transmission. Through the use of cross-model initialization and fusion loss optimization to achieve co-training of the SNN model, the SNN model can be used for EEG emotion recognition, significantly improving the accuracy and stability of EEG emotion recognition.

[0096] Those of ordinary skill in the art will realize that the embodiments described herein are for the purpose of assisting the reader in understanding the principles of the present invention and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, various changes and modifications can be made to the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. An EEG emotion recognition method based on a spiking neural network, the specific steps are as follows: S1, collect EEG data and segment the EEG data in the time direction; Collect an EEG signal X C×N , it is divided into T segments in the time direction, then the EEG data expression is as follows: in, C represents the number of channels, and N represents the number of sampling points; S2, build ANN and SNN models with the same network structure; S21, construct ANN model; The ANN model proposes a new pulse neural network structure for time series data processing, including: a fully connected structure FC, a one-dimensional convolution structure Conv1d; Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories; the convolution kernel size of the one-dimensional convolution structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories; The ANN model is built using ReLU neurons or Sigmoid neurons, using the EEG data obtained in step S1 As input data, the output of the model is ANN The expression is as follows: Among them, in the ANN model, the trainable parameter is the model weight W ANN ; S22. Use the improved LIF model to build the SNN model; The SNN model and the ANN model in step S21 have exactly the same network structure; the SNN model uses the improved LIF spiking neurons to collaboratively learn the biological parameters of the improved LIF model, that is, to integrate the intrinsic parameters with biological rationality into the spiking neurons; then the neuron membrane potential u t The iteration formula is as follows: in t =α t ·in t-1 +I t , Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and pulse emission of the neuron at time t, α t represents the membrane potential attenuation coefficient; then the neuron discharge o t The expression is as follows: o t =H(u t -V th ), Among them, V th represents the neuron discharge threshold, H(·) represents the step activation function; when the neuron membrane potential is greater than the discharge threshold, then o t =1, otherwise o t =0; and the membrane potential attenuation coefficient α t The expression is as follows: Where τ represents the membrane time constant; The input and output of the improved LIF model are pulse sequences, and the trainable parameters include membrane time constant and discharge threshold. The SNN model is constructed by using the improved LIF neuron, and the trainable parameters of the SNN model include: the weight W of the model SNN , discharge threshold V th and membrane time constant τ; using the EEG data obtained in step S1 As input data, the output Y of the SNN model SNN The expression is as follows: Among them, O t represents the discharge pulse of the SNN output layer at time t; S3. Based on step S2, a supervised learning method of an ANN model based on time averaging is proposed. The ANN model is pre-trained using the back propagation method and data labels. When the loss function of the ANN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain a trained ANN model. The output of the ANN model is Y in step S21 ANN , data label Lable EEG is the true category of the EEG data; supervised training is performed using the output of the ANN model and the data label, and the loss function value L ANN The expression is as follows: L ANN =MSE(Y ANN -Lable EEG ), Among them, MSE means minimum mean square error; Finally, the loss function value and back propagation method are used to train the parameters of the ANN model to obtain a trained ANN model, and the weight parameter of the trained ANN model is W ANN ; S4. Based on step S3, collaborative training of SNN model parameters is performed to obtain a trained SNN model; S41, cross-model initialization; Using the same structure between ANN and SNN models, the weights W of the ANN model trained in step S3 are used. ANN Initialize the weights of the SNN model, that is, W ANN Directly migrate to the SNN model; at the same time, the biological parameters of the SNN model, membrane time constant τ and discharge threshold V th Initialize; S42, mixed loss optimization; The prediction result Y using the trained ANN model ANN and EEG data labels EEG Construct a hybrid loss function together and use the hybrid loss function to supervise the training of the SNN model; In the training process of the SNN model, the mixed loss function Loss includes: information loss L info and label loss L label , the expression is as follows: Loss=L label +W info ·L info , Among them, W info represents the weight of the control information constraint, L label Represents the true label and the SNN model prediction value Y SNN Mean square error MSE between; information loss L info Based on the information label info_label generated by ANN output and the BISNN predicted value Y SNN The KL divergence between them is calculated; the information label definition expression is as follows: info_label=softmax(Y ANN / temp) Among them, temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model; Based on the mixed loss function value Loss, the weight parameter W of the SNN model is trained collaboratively using the back propagation method. SNN , discharge threshold V th And membrane time constant τ, when the loss function of the SNN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain the trained SNN model.

Images