Electroencephalogram emotion recognition method based on pulse convolutional neural network

Abstract

The application discloses a kind of electroencephalogram emotion recognition methods based on pulse convolutional neural network.The present application is after the preprocessing of electroencephalogram emotion data, it extracts differential entropy feature and time window intercept, then it is encoded using pulse convolutional encoder;Pulse convolution feature extractor extracts pulse data features;Pulse fully connected classifier learns and classifies.The present application directly trains the pulse neural network by fusing the pulse convolution layer and the pulse fully connected layer, achieves the purpose of electroencephalogram emotion classification, and transmits pulses between layers and time slices in the network during the training process through the pulse convolution layer, extracts the space-time information in electroencephalogram data, and then classifies emotions through the feature learning of the pulse fully connected layer.The present application combines the existing traditional neural network with the pulse neuron by using the characteristics of pulse neural network, improves the performance and generalization ability of the model, and realizes good emotion recognition effect and smaller model size.

Description

Technical Field

[0001] This invention relates to the field of brainwave emotion recognition technology in the field of biometrics, and mainly to a brainwave emotion recognition method based on a pulsed convolutional neural network. Background Technology

[0002] Electroencephalogram (EEG) data is obtained by amplifying and recording spontaneous bioelectrical potentials in the brain through sophisticated electronic devices applied to the scalp. EEG data records a wealth of information about the activity of brain neurons, including rich spatiotemporal information, which can be used to assess a subject's emotions. In recent years, an increasing number of studies have focused on emotion recognition, bridging the gap between humans and computers.

[0003] Spiking Neural Networks (SNNs) contain neurons with temporal dynamics and low-power binary pulse propagation patterns. They highly borrow from the physical characteristics and learning patterns of the biological brain, thus possessing powerful capabilities in spatiotemporal information representation, asynchronous event information processing, and low-power learning. Furthermore, the cross-integration of SNNs with current computer science-driven artificial neural networks, such as deep convolutional networks, is considered a powerful approach to developing artificial general intelligence. Given the diminishing returns of deep convolutional networks and the increasingly questionable rationality of their underlying architecture, the fusion and complementary development of SNNs and deep artificial neural networks is gradually becoming a trend for the next generation of artificial neural networks.

[0004] EEG data contains rich spatial information from different acquisition points and continuous temporal information from high-frequency sampling within a time period. On the one hand, traditional artificial neural network EEG data processing and classification methods, such as convolutional neural networks (CNNs) and graph convolutional neural networks (GCNs), extract spatial information from EEG data, and then use methods such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) to extract temporal information for classification. These methods have achieved good results in emotion recognition accuracy, but the high cost of multiplication calculations and the excessively large model size make it difficult to widely implement in portable embedded systems. On the other hand, while existing spiking neural network models perform well in terms of energy consumption and biological interpretability, there are currently no standardized methods or breakthroughs in accuracy, resulting in significant shortcomings in the accuracy of EEG emotion classification.

[0005] Based on the above technical characteristics, and compared with existing traditional artificial neural network recognition methods, this invention achieves a balance between network performance and size by integrating spiking neural networks with existing convolutional neural networks. This enables effective training and recognition of various types of EEG emotion data, and can be integrated into more efficient and portable neuromorphic hardware and embedded devices. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a brainwave emotion recognition method based on a spiking convolutional neural network. This invention integrates existing traditional artificial neural network methods for brainwave emotion recognition with spiking neural networks. It directly trains a spiking neural network that incorporates spiking convolutional layers and spiking fully connected layers to achieve the goal of classifying brainwave emotions. During training, spiking convolutional layers transmit pulses between layers and time slices of the network, extracting spatiotemporal information from the brainwave data. Then, the spiking fully connected layers perform feature learning for the emotion classification task. Ultimately, this method achieves a higher accuracy rate in brainwave emotion classification than existing spiking neural networks, and is comparable to current traditional artificial neural networks, while maintaining operational efficiency, thus achieving a balance between operational efficiency and performance.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0008] Step 1: Data Collection: Collect EEG data of subjects while watching movie clips, and immediately after watching each movie clip, ask subjects to complete a questionnaire to report their emotional response to each movie clip. The emotional response includes positive, neutral, and negative. The EEG data are EEG signals from 32 channels at designated electrode locations collected through a 10-20 international standard lead system.

[0009] Step 2, Data Preprocessing: The original EEG signal is downsampled and preprocessed to remove ocular artifacts. The time-domain signal is filtered using a Hamming window and subjected to a Fast Fourier Transform. The differential entropy (DE) characteristics of 32 channels in 4 frequency bands of all acquisition electrodes are calculated by using a sliding window through the channel signal of each acquisition electrode.

[0010] Step 3, Sample Generation: The processed differential entropy features are processed into a non-overlapping sliding window according to a time window T. This operation is performed for each channel and each frequency band, resulting in: Time * Channel * Frequency Band. The one-dimensional channel data sequence is then converted into a two-dimensional network matrix sequence, where the positional correspondence is obtained through the two-dimensional topological map of the EEG electrode cap. The final input sample is: Time * H * W * Frequency Band; where H and W are the height and width of the two-dimensional network matrix.

[0011] Step 4: Define the model input and output: A single training iteration of the model inputs a batch of samples, where each sample's structure is time * H * W * frequency band. Therefore, the model input structure is Input = batch * time * H * W * frequency band. The model output structure is a set of one-hot vectors, with the structure Output(batch * classes), where classes represent the probability that the sample belongs to that class, expressed as a decimal from 0 to 1.

[0012] Step 5: Define the spiking neuron:

[0013] As the most basic unit of a spiking neural network, the spiking neuron controls the generation and propagation of signals within the network. The neuron model used in this invention is the LIF (Leaky Integrity Fire) model, and its dynamic equations can be used as follows:

[0014]

[0015] X t =wI(t)

[0016] Formula (1), where H t and V t X represents the membrane potential after the neuron's dynamic changes and the membrane potential after the time-step trigger pulse, respectively. t Indicates external input, when V t-1 =0. Integration progress. This allows LIF neurons to remember the current input information, and This can be viewed as the forgetting of some information from the past. This formula shows that the balance between memory and forgetting is controlled by the membrane time constant c = 1.2.

[0017] S t =Θ(H t -V th (2)

[0018]

[0019] Formula (2), S t The output pulse at time t represents the pulse generation process, where Θ(x) is the Heaviside step function, defined as: Θ(x) = 1 when x ≥ 0, and Θ(x) = 0 when x < 0.

[0020] V t =H t (1-S t )+V reset S t (3)

[0021] Equation (3) describes the return of the membrane potential to V after the pulse is generated. reset The process is a hard reset type, which is more suitable for the deep spiking neural network in this model.

[0022] This model is a spiking neural network that combines traditional convolutional neural networks, fully connected networks, and spiking neuron mechanisms. By modifying the activation function and the activation function of the aforementioned spiking neuron activation mode, and supplementing it with a gradient substitution function, the network training process enables layers to input and output through the aforementioned activation mode, and the time slices within a layer to sequentially output, activate, and transmit pulses, thus achieving better extraction and learning of temporal information.

[0023] Step 6: Define the spiking neural network architecture:

[0024] The spiking neural network structure consists of three parts: an adaptive spiking convolutional encoder, a spiking convolutional feature extraction network, and a spiking fully connected classifier.

[0025] The adaptive pulse convolutional encoder, as the top layer of the spiking neural network structure, aims to learn to convert real-valued sample data into binary output in the form of pulses through pulse convolution. Subsequent spiking neural network structures will use pulse values ​​as data for processing and classification.

[0026] The pulse convolution feature extraction network learns and extracts effective temporal and spatial information from EEG sample data through the interaction of outputs between layers and the sequential interaction of temporal segments between layers in a multi-layer pulse convolutional neural network. This enables the extracted pulse feature information to be effectively used for emotion classification and recognition.

[0027] The spiking fully connected classifier learns the final spiking neuron weights through the interaction of layer-to-layer outputs and the sequential interaction of time segments between layers in a two-layer spiking fully connected network. Then, the final spiking output is obtained. Here, due to the binarization of the spiking output, directly using the spiking output as the classification result would greatly reduce the model's generality and robustness. Therefore, the output of the final spiking fully connected layer in this invention will be 10 times the number of classification categories. Then, average pooling is used to perform a 10-to-1 voting process to finally obtain the classification categories.

[0028] Step 7, Define the objective function: The objective function used in training the spiking neural network is the mean squared error function. In the mean squared error function, x represents the input data sample, y represents the output label generated by the training model, and i represents each value in the sample matrix. Its formula (4) is as follows:

[0029] MSE(x i y i )=(x i -y i ) 2 (4)

[0030] Step 8, Training and Testing: Input the training set into the spiking neural network according to the input structure defined in Step 3 for multiple rounds of training. After each round of training, input the trained model into the network using the test set in the same way for prediction. Compare the prediction results as output with the true labels to calculate the mean squared error. Finally, obtain the overall classification accuracy of the spiking neural network on the test set.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention proposes an emotion recognition method based on spiking convolutional neural networks. It efficiently combines the spiking neuron mechanism with traditional convolutional neural networks and fully connected neural networks, which can more effectively extract the temporal information of EEG data. The model has strong generalization ability and is applicable to a variety of datasets. The emotion recognition accuracy is improved compared with existing spiking neural networks. Moreover, the model has significantly reduced running speed and size compared with traditional neural networks of the same structure. It has broad application prospects in neuromorphic hardware and portable embedded devices. Attached Figure Description

[0033] To more clearly illustrate the embodiments and implementation schemes of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart of an EEG emotion classification method based on a spiking neural network provided by the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the structural definition and internal neurodynamics of the LIF (Leaky Integrity Fire) neuron model provided by the present invention.

[0036] Figure 3 A schematic diagram of an EEG emotion classification network structure based on a spiking neural network is provided for the implementation of this invention;

[0037] Figure 4 The experimental performance of the EEG emotion classification method based on spiking neural networks provided by this invention is compared with other existing spiking neural network methods. Detailed Implementation

[0038] The present invention will now be described more clearly and completely with reference to the accompanying drawings and embodiments, so that the advantages and features of the present invention can be more easily understood by those skilled in the art. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0039] Step 1: Collection of raw data:

[0040] First, participants wore a 32-electrode EEG acquisition headgear designed based on the international 10-20 standard. Fifteen film clips (containing positive, neutral, and negative emotions) were selected from a video material library as the emotional stimuli used in the experiment. Participants watched 15 clips per experiment, with a 5-second prompt before each clip. Each clip was presented for 4 minutes, followed by a 45-second self-assessment after each clip, and a 15-second rest period after each clip. The self-assessment phase involved completing a questionnaire to provide feedback on the participants' emotional response to each film clip. The data was down-converted to 200Hz using the EEGLAB tool to remove interference from electrooculography (EOG) and electromyography (EMG) signals. A 0-7Hz bandpass filter was then applied for filtering, and the corresponding data were labeled with emotion tags (-1 for negative, 0 for neutral, and +1 for positive). Finally, the data was saved to a .mat file.

[0041] Step 2, Data Preprocessing and Feature Extraction:

[0042] DE features were extracted for each channel from four frequency bands: theta band (4-7Hz), alpha band (8-13Hz), beta band (14-30Hz), and gamma band (31-50Hz). The specific method is as follows:

[0043] The original data is filtered using a non-overlapping sliding window. A Fast Fourier Transform is performed on the data every second, and the differential entropy of the above four frequency bands is calculated. The differential entropy is defined as follows:

[0044] Differential entropy DE is Shannon information entropy -∑ x ap(x)log(p(x)) is a generalized form of entropy for continuous variables.

[0045]

[0046] Where p(x) represents the probability density function of continuous information, [a, b] represents the range of values ​​for EEG data, and for a given length, it approximately follows a Gaussian distribution N(μ, σ). 2 The differential entropy of the EEG data is:

[0047]

[0048] The processed DE EEG data are then normalized using z-score, where the normalization formula (3) is shown in the figure:

[0049]

[0050] Where X represents the EEG signal on each channel. S is the average value of the EEG signal on each channel, and S is the standard deviation of the EEG signal on each channel.

[0051] Step 3, Sample Generation:

[0052] The processed 0.5s window DE features are then subjected to a non-overlapping sliding window operation within a T=32 time window. This operation is performed on both 32 channels and 4 frequency bands, resulting in a sequence of 32*32*4 (time*channel*frequency band). The one-dimensional channel data sequence is then converted into a 9*9 two-dimensional mesh matrix sequence. The final input sample is then 32*9*9*4 (time*H*W*frequency band).

[0053] Step 4: Define the input and output of the spiking neural network model:

[0054] The input for a single training iteration of the spiking neural network model consists of a batch of 16 samples. Each sample has the structure given in step 3 (32*9*9*4), therefore the input structure of the spiking neural network is Input = 16*32*9*9*4. The output structure of the spiking neural network is a set of 16 one-hot vectors, with the structure Output = batch * classes, where classes represents the probability of each bit belonging to that class, expressed as a decimal from 0 to 1.

[0055] Step 5: Define the spiking neuron:

[0056] Spiking neurons, as the most basic unit of spiking neural networks, control the generation and propagation of signals in the network. The neuron model used in this invention is the LIF (Leaky Integrity Fire) model, and its model dynamic equations can be used as follows:

[0057]

[0058] X t =wI(t)

[0059] Formula (4), where H t and V t X represents the membrane potential after the neuron's dynamic changes and the membrane potential after the time-step trigger pulse, respectively. t Indicates external input, when V t-1 =0. Integration progress. This allows LIF neurons to remember the current input information, and This can be viewed as the forgetting of some past information. This formula shows that the balance between memory and forgetting is controlled by the membrane time constant c = 1.2, where the membrane time constant of the pulsed convolutional layer in this model is c = 1.2, and the pulsed fully connected layer is c = 1.2.

[0060] S t =Θ(H t -V th (5)

[0061]

[0062] Formula (5), S t The output pulse at time t represents the pulse generation process, where Θ(x) is the Heaviside step function, defined as: Θ(x) = 1 when x ≥ 0, and Θ(x) = 0 when x < 0.

[0063] V t =H t (1-S t )+V reset S t (6)

[0064] Equation (6) describes the return of the membrane potential to V after the pulse is generated. reset The process is a hard reset type, which is more suitable for the deep spiking neural network in this model.

[0065] To solve the output of the spiking neuron, i.e., in formula (5), its derivative, by definition, is the impulse function:

[0066]

[0067] Directly using the impulse function for gradient descent would obviously make the network training unstable. Therefore, this invention also introduces the gradient substitution function commonly used in directly trained spiking neural networks. The principle is that y = Θ(x) is used during forward propagation, while a different function is used during backward propagation. σ(x) is the substitution function, which is a function with a shape similar to Θ(x), but smooth and continuous. The substitution function used in this invention is the Surrogate arctanx function, defined as follows:

[0068] σ(x, α) = arctan(αx) (8)

[0069] The α parameter is an adjustable parameter used to control the gradient, and is set to α = 2.0.

[0070] Step 6: Define the spiking neural network model architecture:

[0071] An adaptive pulse convolutional encoder is used as the top layer of the model to receive input data, which is then output to the main body of the spiking neural network model. The main body of the model consists of a pulse convolutional feature extraction network and a pulse fully connected classifier, which realizes the extraction and classification of temporal and spatial features of EEG emotion data. The specific implementation is as follows:

[0072] Step 6-1: Define an adaptive pulse convolution encoder as the top layer of the network. Its purpose is to learn to convert real-valued sample data into a suitable pulse-form binary output through pulse convolution. Its structure includes:

[0073] ① Pulsed convolutional layer 1, with the following parameters: number of input channels = 4; number of output channels = 64; kernel size = (3, 3); padding = 1; bias = False;

[0074] ②BatchNorm2d; Normalization is based on formula (10) as follows:

[0075]

[0076] eps: A value added to the denominator for calculation stability; the default value is 1e. -5 ;

[0077] gamma and beta: normalization coefficients, which are 1 and 0 by default and are not added.

[0078] mean(x) and Var(x): the mean and variance of normalized data;

[0079] The activation layer of the 3LIF neuron has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx.

[0080] Step 6-2: Define a pulse convolutional feature extraction network. Through the interaction of layer-to-layer outputs and the sequential interaction of temporal segments between layers in a multi-layer pulse convolutional neural network, effective temporal and spatial information from EEG sample data is learned and extracted. Its structure includes:

[0081] 1. Pulsed convolutional layer 2, with the following parameters: number of input channels = 64; number of output channels = 128; kernel size = (3,3); padding = 1; bias = False;

[0082] BatchNorm2d;

[0083] The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx.

[0084] 2 pulsed convolutional layers 3, with the following parameters: number of input channels = 128; number of output channels = 256; kernel size = (3,3); padding = 1; bias = False;

[0085] BatchNorm2d;

[0086] The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx.

[0087] 3 pulsed convolutional layers 4, with the following parameters: number of input channels = 256; number of output channels = 64; kernel size = (1,1); bias = False;

[0088] BatchNorm2d;

[0089] The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx.

[0090] Step 6-3: Define a spiking fully connected classifier. The final spiking neuron weights are obtained through the interaction of layer-to-layer outputs and the sequential interaction of temporal segments between two spiking fully connected layers, ultimately yielding the classification output. Its structure consists of two spiking fully connected layers and one average pooling voting layer, as shown below:

[0091] 1. Dropout, with the parameter: p = 0.25;

[0092] Pulse fully connected layer 1, with the following parameters: Number of input channels = 64*9*9; Number of output channels = 64*6*6; bias = False;

[0093] The LIF neuron activation layer has the following parameters: membrane time constant = 2.0; gradient substitution function = Surrogatearctanx.

[0094] ②Dropout, with the parameter: p = 0.25;

[0095] Pulse fully connected layer 2, with the following parameters: number of input channels = 64 * 6 * 6; number of output channels = 20; bias = False;

[0096] The LIF neuron activation layer has the following parameters: membrane time constant = 2.0; gradient substitution function = Surrogatearctanx.

[0097] ③ Average pooling voting layer, with the following parameters: voting ratio = 10;

[0098] Step 7, Define the objective function: The objective function used in training the spiking neural network is the mean squared error function (MSE Loss). In the mean squared error function, x represents the input data sample, y represents the output label generated by the training model, and i represents each value in the sample matrix. Its formula (4) is as follows:

[0099] MSE(x i y i )=(x i -y i ) 2 (10)

[0100] Step 8: Model Training:

[0101] The training process for the pulsed convolutional layer and pulsed fully connected layer described in step 6, for a set of inputs (16*32*9*9*4), can be summarized as follows: First, the input (16*32*9*9*4) is transposed to (32*16*9*9*4) and (T*batch*H*W*frequency band) for time-step simulation of the spiking neural network. Simultaneously, an entire batch of data can be calculated to improve efficiency. Then, during training, the pulsed time-step simulation is performed as follows: Figure 2 As shown, the left i-1 node represents the state of the LIF neuron at time i-1, and the right i node represents the state of the LIF neuron at time i. Right neuron H t The membrane potential is from V on the left. t-1 Charging. When H t-1 Input X is obtained at t-1. t-1 At that time, according to formula (1), H t-1 If the potential changes and exceeds the threshold, a pulse S is generated according to formula (1). t-1 According to formula (3), H t-1 Reset to the membrane potential V after triggering t-1 If no pulse is generated, V can be obtained directly. t-1 The generated pulse will then become the input to the next layer, i.e., the triggered membrane potential V. t-1 The pre-excitation potential H will become the next time step t-1 This process is performed sequentially on all nodes in a layer to enable adjacent spiking neuron nodes to learn and transmit temporal information.

[0102] Step 9: Model Testing and Result Feedback:

[0103] After the above steps, the trained model is obtained. The input EEG test dataset is fed into the trained spiking neural network model to obtain the emotion classification results of the EEG data. The best result of each round of tests is selected and saved. Then, the final test accuracy is obtained by averaging the experimental data of each of the 32 subjects in each round.

[0104] Step 10, Result Evaluation:

[0105] To verify the effectiveness of this invention, experimental data on three categories of emotions from 32 participants were selected: Valence, Arousal, and Dominance. A spiking neural network was used for binary classification of emotions to evaluate the feature extraction performance of the model. Experimental verification is shown in the attached figure. Figure 4 As shown, compared with other methods, the accuracy of the method of the present invention reached 94.56%, 94.81%, and 90.73%, respectively, demonstrating that the present invention has a better and more generalizable effect in extracting image features from EEG data, and the classification accuracy in emotion recognition is also significantly higher than other existing spiking neural network methods.

Claims

1. A brainwave emotion recognition method based on a pulsed convolutional neural network, characterized in that... Includes the following steps: Step 1: Data Collection: Collect EEG data of subjects while watching movie clips, and immediately after watching each movie clip, ask subjects to complete a questionnaire to report their emotional response to each movie clip. The emotional response includes positive, neutral, and negative. The EEG data are EEG signals from 32 channels at designated electrode locations collected through a 10-20 international standard lead system. Step 2, Data Preprocessing: The original EEG signal is downsampled and preprocessed to remove ocular artifacts. The time-domain signal is filtered using a Hamming window and subjected to a fast Fourier transform. The differential entropy characteristics of 32 channels in 4 frequency bands of all acquisition electrodes are calculated by using a sliding window through the channel signal of each acquisition electrode. Step 3, Sample generation: The processed differential entropy features are processed into a non-overlapping sliding window according to the T time window. This operation is performed for each channel and each frequency band to obtain: time * channel * frequency band. The one-dimensional channel data sequence is then converted into a two-dimensional network matrix sequence, where the positional correspondence is obtained through the two-dimensional topological map of the EEG electrode cap; the final input sample is: time * H * W * frequency band; where H and W are the height and width of the two-dimensional network matrix. Step 4: Define the model input and output: The input of a single training iteration of the model contains a set of samples, where the structure of each sample is time * H * W * frequency band. Therefore, the model input structure is Input = batch * time * H * W * frequency band. The model output structure is a set of one-hot vectors, and the output structure is Output = batch * classes, where classes represent the probability that the sample belongs to this class, represented by a decimal from 0 to 1; batch represents the number of samples in a set. Step 5: Define the spiking neuron: The LIF model is used as the neuron model; the spiking neural network, which combines the traditional convolutional neural network, the fully connected network and the spiking neuron mechanism, is modified by the activation function and the activation function of the above-mentioned spiking neuron activation mode, supplemented by the gradient substitution function, so that the network training process can achieve input and output between layers through the firing mode, and the output, firing and transmission of pulses are also performed sequentially in the time slices within the layer. The model's dynamic equations include: X t =wI(t) Formula (4), where H t and V t These represent the membrane potential after dynamic changes in the neuron and the membrane potential after a time-step trigger pulse, respectively; X t Indicates external input, when V t-1 =0; Integration progress This allows LIF neurons to remember the current input information, and This can be seen as forgetting some information from the past; the formula shows that the balance between memory and forgetting is controlled by the membrane time constant c = 1.2, where the membrane time constant of the pulsed convolutional layer in the model is c = 1.2 and the pulsed fully connected layer is c = 1.

2. Step 6: Define the spiking neural network architecture: The spiking neural network architecture includes an adaptive spiking convolutional encoder, a spiking convolutional feature extraction network, and a spiking fully connected classifier; An adaptive pulse convolution encoder is defined as the top layer of the network, designed to learn how to convert real-valued sample data into a suitable pulse-form binary output through pulse convolution. Its structure includes: (1) Pulsed convolutional layer 1, with the following parameters: number of input channels = 4; number of output channels = 64; kernel size = (3,3); padding = 1; bias = False; (2) BatchNorm2d; Normalization is based on formula (10) as follows: eps: A value added to the denominator for calculation stability; the default value is 1e. -5 ; gamma and beta: normalization coefficients, which are 1 and 0 by default and are not added. mean(x) and Var(x): the mean and variance of normalized data; (3) LIF neuron activation layer, with the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx; Step 7: Define the objective function: The objective function used during the training of the spiking neural network is the mean squared error function; Step 8, Training and Testing: Input the training set into the spiking neural network according to the input structure defined in Step 3 for multiple rounds of training; after each round of training, input the trained model into the network using the test set in the same way to make predictions, use the prediction results as outputs to compare with the true labels to calculate the mean squared error, and finally obtain the overall classification accuracy of the spiking neural network on the test set.

2. The EEG emotion recognition method based on a pulsed convolutional neural network according to claim 1, characterized in that... The neuron model used is the LIF model, and its model dynamics equations also include: S t =Θ(H t -V th ) (5) Formula (5), S t The output pulse at time t represents the pulse generation process, where Θ(x) is the Heaviside step function, defined as: Θ(x) = 1 when x ≥ 0, and Θ(x) = 0 when x < 0; V t =H t (1-S t )+V reset S t (6) Equation (6) describes the return of the membrane potential to V after the pulse is generated. reset The process is a hard reset. To solve the output of the spiking neuron, i.e., in formula (5), its derivative, by definition, is the impulse function: Directly using the impulse function for gradient descent would obviously make the network training extremely unstable. Therefore, a gradient substitution function for directly trained spiking neural networks is introduced: y = Θ(x) is used during forward propagation, while a different function is used during backpropagation. σ(x) is a substitution function, which is a function with a shape similar to Θ(x), but smooth and continuous. The substitution function used is the Surrogate arctanx function, which is defined as follows: σ(x, α) = arctan(αx) (8) The α parameter is an adjustable parameter used to control the gradient, and is set to α = 2.

0.

3. The EEG emotion recognition method based on a pulsed convolutional neural network according to claim 2, characterized in that... An adaptive pulse convolutional encoder is used as the top layer of the model to receive input data, which is then output to the main body of the spiking neural network model. The main body of the model consists of a pulse convolutional feature extraction network and a pulse fully connected classifier, which realizes the extraction and classification of temporal and spatial features of EEG emotion data. The specific implementation is as follows: A pulsed convolutional feature extraction network is defined. Through the interaction of layer-to-layer outputs and the sequential interaction of temporal segments between layers in a multi-layered pulsed convolutional neural network, effective temporal and spatial information is learned and extracted from EEG sample data. Its structure includes: (1) Pulsed convolutional layer 2, with the following parameters: number of input channels = 64; number of output channels = 128; kernel size = (3,3); padding = 1; bias = False; BatchNorm2d; The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx; (2) Pulsed convolutional layer 3, with the following parameters: number of input channels = 128; number of output channels = 256; kernel size = (3,3); padding = 1; bias = False; BatchNorm2d; The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx; (3) Pulsed convolutional layer 4, with the following parameters: number of input channels = 256; number of output channels = 64; kernel size = (1,1); bias = False; BatchNorm2d; The LIF neuron activation layer has the following parameters: membrane time constant = 1.2; gradient substitution function = Surrogatearctanx; A spiking fully connected classifier is defined. It learns the final spiking neuron weights through the interaction of layer-to-layer outputs and the sequential interaction of time segments between layers in a two-layer spiking fully connected network, ultimately yielding the classification output. Its structure consists of two spiking fully connected layers and one average pooling voting layer, as shown below: (1) Dropout, with the parameter: p = 0.25; Pulse fully connected layer 1, with the following parameters: Number of input channels = 64*9*9; Number of output channels = 64*6*6; bias = False; The LIF neuron activation layer has the following parameters: membrane time constant = 2.0; gradient substitution function = Surrogatearctanx; (2) Dropout, with the parameter: p = 0.25; Pulse fully connected layer 2, with the following parameters: number of input channels = 64 * 6 * 6; number of output channels = 20; bias = False; The LIF neuron activation layer has the following parameters: membrane time constant = 2.0; gradient substitution function = Surrogatearctanx; (3) Average pooling voting layer, with the following parameters: voting ratio = 10.

4. The EEG emotion recognition method based on a pulsed convolutional neural network according to claim 2 or 3, characterized in that... The training process of the spiking convolutional layer and spiking fully connected layer described in step 6 for a set of inputs can be summarized as follows: First, the input 16*32*9*9*4 is transposed to become 32*16*9*9*4 for time-step simulation of the spiking neural network, while simultaneously calculating the entire batch of data to improve efficiency; then, the spiking time-step simulation during training is as follows: the left i-1 node is denoted as the state of the LIF neuron node at i-1, and the right i node represents the state of the LIF neuron at i; the right neuron H t The membrane potential is from V on the left. t-1 Charging; when H t-1 Input X is obtained at t-1. t-1 At that time, according to formula (1), H t-1 If the potential changes and exceeds a threshold, a pulse S is generated. t-1 According to formula H t-1 Reset to the membrane potential V after triggering t-1 If no pulse is generated, V is obtained directly. t-1 The generated pulse then becomes the input to the next layer, i.e., the membrane potential V after triggering. t-1 The pre-excitation potential H will become the next time step t-1 This process is performed sequentially on all nodes in a layer to enable adjacent spiking neuron nodes to learn and transmit temporal information.

Images