EEG emotion recognition method based on spiking neural network

Abstract

The invention discloses an EEG emotion recognition method based on a spiking neural network, which comprises the following steps: firstly collecting EEG data and segmenting the EEG data in a time direction, then constructing an ANN model and an SNN model which have the same network structure, inputting the EEG data to the ANN model for training, finishing the training of the ANN model after the precision of the ANN model subjected to back propagation training is stable, and finally obtaining the EEG emotion recognition result. And finally, performing weight initialization on the SNN model by using the trained ANN model weight, cooperatively training the weight and internal parameters of the SNN by using mixed loss and back propagation, and when the precision of the model is stable, ending the training to obtain the trained SNN model. According to the method, internal parameters with biological rationality are integrated into spiking neurons, so that information processing is optimized and signal transmission is stabilized on the neuron level, collaborative training of the SNN model is realized by using cross-model initialization and fusion loss optimization, the SNN model can be used for EEG emotion recognition, and the recognition efficiency is improved. And the precision and the stability of EEG emotion recognition are obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotional computing and brain-like intelligent application technology, and specifically relates to an EEG emotion recognition method based on a pulse neural network. Background Art

[0002] Accurately detecting and decoding emotional states through physiological signals is a key challenge in the design and development of autonomous intelligent brain-computer interfaces (BCIs). Among the many physiological signals, electroencephalogram (EEG) has become a promising direction for achieving reliable emotion recognition due to its rich temporal dynamics and ability to capture real-time neural activities.

[0003] In recent years, some models based on spiking neural networks (SNNs) have been proposed for analyzing EEG signals. These models take advantage of the advantages of SNNs in time series data processing, and their performance does not rely on complex architecture design and information interaction. Despite certain progress, there are still many challenges to further improve the performance of SNN models. Previous studies have successfully applied the gradient approximation function to the training of SNN models by introducing it into the back-propagation algorithm. This method effectively alleviates the non-differentiability problem of spike activity, thereby reducing the difficulty of model training. Another widely adopted strategy is to convert the pre-trained artificial neural network (ANN) model into an SNN, avoiding the complexity of directly training the SNN model. However, these methods usually ignore the rich intrinsic properties of biological neurons and fail to fully explore the information and knowledge contained in the pre-trained ANN model. Summary of the invention

[0004] To solve the above technical problems, the present invention provides an EEG emotion recognition method based on spiking neural network, which uses cross-model initialization and fusion loss optimization to realize collaborative training of SNN models, significantly improving the accuracy and stability of EEG emotion recognition.

[0005] The technical solution adopted by the present invention is: an EEG emotion recognition method based on a spiking neural network, and the specific steps are as follows:

[0006] S1, collect EEG data and segment the EEG data in the time direction;

[0007] Collect an EEG signal X C×N , it is divided into T segments in the time direction, then the EEG data expression is as follows:

[0008]

[0009] Among them, C represents the number of channels and N represents the number of sampling points.

[0010] S2, build ANN and SNN models with the same network structure;

[0011] S21, construct ANN model;

[0012] The ANN model proposes a new pulse neural network structure for time series data processing, including: a fully connected structure FC and a one-dimensional convolution structure Conv1d.

[0013] Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories. The convolution kernel size of the one-dimensional convolution structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories.

[0014] The ANN model is built using ReLU neurons or Sigmoid neurons, using the EEG data obtained in step S1 As input data, the output of the model is ANN The expression is as follows:

[0015]

[0016] Among them, in the ANN model, the trainable parameter is the model weight W ANN .

[0017] S22. Use the improved LIF model to build the SNN model;

[0018] The SNN model and the ANN model in step S21 have exactly the same network structure. The SNN model uses the improved LIF spiking neuron to collaboratively learn the biological parameters of the improved LIF model, that is, to integrate the intrinsic parameters with biological rationality into the spiking neuron; then the neuron membrane potential u t The iteration formula is as follows:

[0019] u t =α t ·u t-1 +I t ,

[0020] Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and pulse emission of the neuron at time t, α t represents the membrane potential attenuation coefficient. Then the neuron discharges o t The expression is as follows:

[0021] o t =H(u t -V th ),

[0022] Among them, V threpresents the neuron discharge threshold, and H(·) represents the step activation function. When the neuron membrane potential is greater than the discharge threshold, then o t =1, otherwise o t = 0. And the membrane potential attenuation coefficient α t The expression is as follows:

[0023]

[0024] Here, τ represents the membrane time constant.

[0025] The input and output of the improved LIF model are pulse sequences, and the trainable parameters include membrane time constant and discharge threshold. When the improved LIF neuron is used to build an SNN model, the trainable parameters of the SNN model include: the model weight W SNN , discharge threshold V th and membrane time constant τ. Using the EEG data obtained in step S1 As input data, the output Y of the SNN model SNN The expression is as follows:

[0026]

[0027] Among them, O t Represents the discharge pulse of the SNN output layer at time t.

[0028] S3. Based on step S2, a supervised learning method of an ANN model based on time averaging is proposed. The ANN model is pre-trained using the back propagation method and data labels. When the loss function of the ANN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain a trained ANN model.

[0029] The output of the ANN model is Y in step S21 ANN , data label Lable EEG is the true category of the EEG data. The output of the ANN model and the data label are used for supervised training, and the loss function value L ANN The expression is as follows:

[0030] L ANN =MSE(Y ANN -Label EEG ),

[0031] Here, MSE stands for minimum mean square error.

[0032] Finally, the loss function value and back propagation method are used to train the parameters of the ANN model to obtain a trained ANN model, and the weight parameter of the trained ANN model is W ANN .

[0033] S4. Based on step S3, collaborative training of SNN model parameters is performed to obtain a trained SNN model;

[0034] S41, cross-model initialization;

[0035] Using the same structure between ANN and SNN models, the weights W of the ANN model trained in step S3 are used. ANN Initialize the weights of the SNN model, that is, W ANN Directly migrate to the SNN model. At the same time, the biological parameters of the SNN model, membrane time constant τ and discharge threshold V th Initialize;

[0036] S42, mixed loss optimization;

[0037] The prediction result Y using the trained ANN model ANN and EEG data labels EEG Together, we construct a hybrid loss function and use it to supervise the training of the SNN model.

[0038] In the training process of the SNN model, the mixed loss function Loss includes: information loss L info and label loss L label , the expression is as follows:

[0039] Loss = L label +W info ·L info ,

[0040] Among them, W info represents the weight of the control information constraint, L label Represents the true label and the SNN model prediction value Y SNN The mean square error MSE between them. Information loss L info Based on the information label info_label generated by ANN output and the BISNN predicted value Y SNN The KL divergence between them is calculated. The information label definition expression is as follows:

[0041] info_label = softmax(Y ANN / temp)

[0042] Among them, temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model.

[0043] Based on the mixed loss function value Loss, the weight parameter W of the SNN model is trained collaboratively using the back propagation method. SNN , discharge threshold V thAnd membrane time constant τ, when the loss function of the SNN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain the trained SNN model.

[0044] Beneficial effects of the present invention: The method of the present invention first collects EEG data and segments the EEG data in the time direction, then constructs ANN and SNN models with the same network structure, inputs EEG data into the ANN model for training, and completes the training of the ANN model when the accuracy of the ANN model trained by back-propagation is stable. Finally, the weights of the SNN model are initialized using the trained ANN model weights, and the weights and intrinsic parameters of the SNN are collaboratively trained using mixed loss and back-propagation. When the accuracy of the model is stable, the training is terminated to obtain a trained SNN model. The method of the present invention integrates biologically reasonable intrinsic parameters (emission threshold and membrane time constant) into spiking neurons, thereby optimizing information processing and stabilizing signal transmission at the neuron level, and realizes collaborative training of the SNN model by using cross-model initialization and fusion loss optimization, so that the SNN model can be used for EEG emotion recognition, significantly improving the accuracy and stability of EEG emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of an EEG emotion recognition method based on a spiking neural network of the present invention.

[0046] Figure 2 Schematic diagram of collaborative learning neurons in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0048] like Figure 1 As shown, a flow chart of an EEG emotion recognition method based on a spiking neural network of the present invention, the specific steps are as follows:

[0049] S1, collect EEG data and segment the EEG data in the time direction;

[0050] Collect an EEG signal X C×N , it is divided into T segments in the time direction, then the EEG data expression is as follows:

[0051]

[0052] Among them, C represents the number of channels and N represents the number of sampling points.

[0053] S2, build ANN and SNN models with the same network structure;

[0054] S21, construct ANN model;

[0055] The network structure in this embodiment is shown in Table 1. The ANN model proposes a new pulse neural network structure for time series data processing (time series signal classification), including: a fully connected structure (FC) and a one-dimensional convolution structure (Conv1d).

[0056] Table 1

[0057]

[0058] Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories. The convolution kernel size of the one-dimensional convolution structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories.

[0059] The ANN model is built using ReLU neurons or Sigmoid neurons, using the EEG data obtained in step S1 As input data, the output of the model is ANN The expression is as follows:

[0060]

[0061] Among them, in the ANN model, the trainable parameter is the model weight W ANN .

[0062] S22. Use the improved LIF model to build the SNN model;

[0063] The SNN model and the ANN model in step S21 have exactly the same network structure. The SNN model uses an improved leaky-integrate and fire (LIF) pulse neuron. The SNN model is built using an improved LIF neuron, and proposes to collaboratively learn the biological parameters of the improved LIF model, that is, to integrate biologically reasonable intrinsic parameters (firing threshold and membrane time constant) into the pulse neuron. The schematic diagram of the collaborative learning neuron in this embodiment is shown in FIG. Figure 2 As shown; the neuron membrane potential u t The iteration formula is as follows:

[0064] u t =α t ·u t-1 +I t ,

[0065] Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and pulse emission of the neuron at time t, αt represents the membrane potential attenuation coefficient. Then the neuron discharges o t The expression is as follows:

[0066] o t =H(u t -V th ),

[0067] Among them, V th represents the neuron discharge threshold, and H(·) represents the step activation function. When the neuron membrane potential is greater than the discharge threshold, then o t =1, otherwise o t = 0. And the membrane potential attenuation coefficient α t The expression is as follows:

[0068]

[0069] Here, τ represents the membrane time constant.

[0070] The input and output of the improved LIF model are pulse sequences, and the trainable parameters include membrane time constant and discharge threshold. When the improved LIF neuron is used to build an SNN model, the trainable parameters of the SNN model include: the model weight W SNN , discharge threshold V th and membrane time constant τ. Using the EEG data obtained in step S1 As input data, the output Y of the SNN model SNN The expression is as follows:

[0071]

[0072] Among them, O t Represents the discharge pulse of the SNN output layer at time t.

[0073] S3. Based on step S2, a supervised learning method of an ANN model based on time averaging is proposed. The ANN model is pre-trained using the back propagation method and data labels. When the loss function of the ANN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain a trained ANN model.

[0074] The output of the ANN model is Y in step S21 ANN , data label Lable EEG is the true category of the EEG data (in this example, a three-category task, the category is defined as 100, 010 and 001). The output of the ANN model and the data label are used for supervised training, and the loss function value L is obtained. ANN The expression is as follows:

[0075] L ANN =MSE(YANN -Label EEG ),

[0076] Here, MSE stands for minimum mean square error.

[0077] Finally, the loss function value and back propagation method are used to train the parameters of the ANN model to obtain a trained ANN model, and the weight parameter of the trained ANN model is W ANN .

[0078] S4. Based on step S3, collaborative training of SNN model parameters is performed to obtain a trained SNN model;

[0079] S41, cross-model initialization;

[0080] Using the same structure between ANN and SNN models, the weights W of the ANN model trained in step S3 are used. ANN Initialize the weights of the SNN model, that is, W ANN Directly migrate to the SNN model. At the same time, the biological parameters of the SNN model, membrane time constant τ and discharge threshold V th Initialization is performed. In this embodiment, the membrane time constant τ and the discharge threshold V th They are set to 2.0 and 1.0 mV respectively;

[0081] S42, mixed loss optimization;

[0082] The prediction result Y using the trained ANN model ANN and EEG data labels EEG Together, we construct a hybrid loss function and use it to supervise the training of the SNN model.

[0083] In the training process of the SNN model, the mixed loss function Loss includes: information loss (L info ) and label loss (L label ), the expression is as follows:

[0084] Loss = L label +W info ·L info ,

[0085] Among them, W info represents the weight of the control information constraint, L label Represents the true label and the SNN model prediction value Y SNN The mean square error (MSE) between them. Information loss L info Based on the information label (info_label) generated by the ANN output and the BISNN predicted value Y SNNThe Kullback-Leibler (KL) divergence between them is calculated. The information label definition expression is as follows:

[0086] info_label = softmax(Y ANN / temp)

[0087] Among them, temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model.

[0088] Based on the mixed loss function value Loss, the weight parameter W of the SNN model is trained collaboratively using the back propagation method. SNN , discharge threshold V th And membrane time constant τ, when the loss function of the SNN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain the trained SNN model.

[0089] This embodiment is further verified by simulation. The SNN model in the method of the present invention is constructed using the two network structures provided in Table 1, and tested on the data of Arousal and Valence. The hyperparameter settings are shown in Table 2, and the experimental results are shown in Table 3.

[0090] Table 2

[0091]

[0092] Table 3

[0093]

[0094] It can be seen from Tables 2 and 3 that, compared with the existing SNN model, the SNN model of the method of the present invention has higher emotion recognition accuracy and more stable results (ie, lower variance).

[0095] In summary, the method of the present invention integrates biologically reasonable intrinsic parameters (firing threshold and membrane time constant) into spiking neurons, thereby optimizing information processing and stabilizing signal transmission at the neuronal level. By using cross-model initialization and fusion loss optimization, the collaborative training of the SNN model is achieved, so that the SNN model can be used for EEG emotion recognition, significantly improving the accuracy and stability of EEG emotion recognition.

[0096] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An EEG emotion recognition method based on a spiking neural network, the specific steps are as follows: S1, collect EEG data and segment the EEG data in the time direction; Collect an EEG signal X C×N , it is divided into T segments in the time direction, then the EEG data expression is as follows: in, C represents the number of channels, and N represents the number of sampling points; S2, build ANN and SNN models with the same network structure; S21, construct ANN model; The ANN model proposes a new pulse neural network structure for time series data processing, including: a fully connected structure FC, a one-dimensional convolution structure Conv1d; Among them, the number of input neurons in the fully connected structure is the same as the number of channels of the EEG data, and the number of output neurons is the same as the number of EEG categories; the convolution kernel size of the one-dimensional convolution structure is the same as the number of channels of the EEG signal, and the number of neurons in the output layer is the same as the number of EEG categories; The ANN model is built using ReLU neurons or Sigmoid neurons, using the EEG data obtained in step S1 As input data, the output of the model is ANN The expression is as follows: Among them, in the ANN model, the trainable parameter is the model weight W ANN ; S22. Use the improved LIF model to build the SNN model; The SNN model and the ANN model in step S21 have exactly the same network structure; the SNN model uses the improved LIF spiking neurons to collaboratively learn the biological parameters of the improved LIF model, that is, to integrate the intrinsic parameters with biological rationality into the spiking neurons; then the neuron membrane potential u t The iteration formula is as follows: in t =α t ·in t-1 +I t , Among them, I t represents the input current of the neuron at time t, u t represents the membrane potential and pulse emission of the neuron at time t, α t represents the membrane potential attenuation coefficient; then the neuron discharge o t The expression is as follows: o t =H(u t -V th ), Among them, V th represents the neuron discharge threshold, H(·) represents the step activation function; when the neuron membrane potential is greater than the discharge threshold, then o t =1, otherwise o t =0; and the membrane potential attenuation coefficient α t The expression is as follows: Where τ represents the membrane time constant; The input and output of the improved LIF model are pulse sequences, and the trainable parameters include membrane time constant and discharge threshold. The SNN model is constructed by using the improved LIF neuron, and the trainable parameters of the SNN model include: the weight W of the model SNN , discharge threshold V th and membrane time constant τ; using the EEG data obtained in step S1 As input data, the output Y of the SNN model SNN The expression is as follows: Among them, O t represents the discharge pulse of the SNN output layer at time t; S3. Based on step S2, a supervised learning method of an ANN model based on time averaging is proposed. The ANN model is pre-trained using the back propagation method and data labels. When the loss function of the ANN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain a trained ANN model. The output of the ANN model is Y in step S21 ANN , data label Lable EEG is the true category of the EEG data; supervised training is performed using the output of the ANN model and the data label, and the loss function value L ANN The expression is as follows: L ANN =MSE(Y ANN -Lable EEG ), Among them, MSE means minimum mean square error; Finally, the loss function value and back propagation method are used to train the parameters of the ANN model to obtain a trained ANN model, and the weight parameter of the trained ANN model is W ANN ; S4. Based on step S3, collaborative training of SNN model parameters is performed to obtain a trained SNN model; S41, cross-model initialization; Using the same structure between ANN and SNN models, the weights W of the ANN model trained in step S3 are used. ANN Initialize the weights of the SNN model, that is, W ANN Directly migrate to the SNN model; at the same time, the biological parameters of the SNN model, membrane time constant τ and discharge threshold V th Initialize; S42, mixed loss optimization; The prediction result Y using the trained ANN model ANN and EEG data labels EEG Construct a hybrid loss function together and use the hybrid loss function to supervise the training of the SNN model; In the training process of the SNN model, the mixed loss function Loss includes: information loss L info and label loss L label , the expression is as follows: Loss=L label +W info ·L info , Among them, W info represents the weight of the control information constraint, L label Represents the true label and the SNN model prediction value Y SNN Mean square error MSE between; information loss L info Based on the information label info_label generated by ANN output and the BISNN predicted value Y SNN The KL divergence between them is calculated; the information label definition expression is as follows: info_label=softmax(Y ANN / temp) Among them, temp represents the temperature parameter, which is used to adjust the smoothness of the softmax output in the ANN model; Based on the mixed loss function value Loss, the weight parameter W of the SNN model is trained collaboratively using the back propagation method. SNN , discharge threshold V th And membrane time constant τ, when the loss function of the SNN model remains unchanged, that is, the accuracy is stable, the model training is stopped to obtain the trained SNN model.

Images