A method for recognizing SSVEP electroencephalogram based on convolution and spiking neural network

Abstract

A kind of SSVEP electroencephalogram recognition method based on convolution and pulse neural network belongs to electroencephalogram recognition method field.The existing non-invasive acquisition electroencephalogram signal exists the problems of low signal recognition quality, or high risk and high cost of high-quality electroencephalogram signal obtained by invasive method.The method of the present application comprises: obtaining and preprocessing SSVEP electroencephalogram signal, then training data set and test data set are built, then SSVEP electroencephalogram signal is pulse coded, then SSVEP electroencephalogram signal is recognized, and first residual module (1), first pulse connection module (2), second residual module (3), second pulse connection module (4) and pulse full connection output module (5) are sequentially arranged;the pulse data output by the above step is subjected to the above pulse neural network, and the SSVEP electroencephalogram signal classification is obtained through the pulse full connection output module (5).Then, subsequent processing is carried out.The method of the present application has low power consumption, reduced operation time of pulse emission, and high robustness of electroencephalogram signal recognition.

Description

Technical Field

[0001] This invention relates to a method for classifying electroencephalogram (EEG) signals, and particularly to an SSVEP EEG signal recognition method based on convolutional and spiking neural networks. Background Technology

[0002] Brain-computer interface (BCI) is a technology that connects the brain's nervous system to electronic devices. By acquiring and processing electroencephalogram (EEG) signals and extracting information from them, it enables the recognition of human intentions. In recent years, non-invasive acquisition of EEG signals from the cerebral cortex has become a major research direction due to its low risk and low cost. Steady-state visual evoked potentials (SSVEP) are a non-invasive and evoked EEG signal research method. Its principle is to stimulate vision with devices that flash at different frequencies (such as flashing LED lights or flashing screen photos), generating corresponding frequency EEG signals in the cerebral cortex, which are then acquired by BCI devices. The electromagnetic waves emitted by brain neurons during stimulation are dispersed and ambiguous, resulting in weak EEG signals. This poses significant challenges to subsequent processing such as stimulus presentation, encoding, and recognition. Furthermore, EEG signal acquisition involves personal privacy and requires the subject's consent for scientific research, making it difficult to obtain large-scale EEG signal samples, thus turning EEG signal recognition into a small-sample problem. Traditional EEG signal recognition networks are large in scale, consume high power, and have slow processing speeds, making them difficult to deploy on portable or wearable devices. To address the challenges of encoding and recognizing weak SSVEP EEG signals, training and generalization capabilities with small samples, and issues related to power consumption, processing speed, and large network size, this invention, based on relevant research, designs an SSVEP EEG signal recognition method based on a combination of convolutional and spiking neural networks. Summary of the Invention

[0003] The purpose of this invention is to address the problems of low signal recognition quality in non-invasive methods and high risk and high cost in invasive methods for obtaining high-quality EEG signals, and to propose an SSVEP EEG signal recognition method based on convolutional and spiking neural networks.

[0004] A method for SSVEP EEG signal recognition based on convolutional and spiking neural networks, the method being implemented through the following steps:

[0005] Step 1, Acquisition and Preprocessing of SSVEP EEG Signals, specifically:

[0006] Low-frequency visual stimulation of 8Hz–30Hz was used, followed by brain-computer interface (BCI) signal acquisition, and then BCI-based data collection. n(n = 0, 1, 2, ..., N) channel EEG signals, which are then segmented into several data segments; among them, 2 n The acquisition channel acquires the number of input channels c, and its data structure is h×w×c; the segmentation operation is specifically based on time t. i Starting with i = 0, 1, 2, ..., n, the EEG data interval is t seconds, and the data length ends with N data points;

[0007] Then, each data segment is stored as an image, i.e., I w×h h is the normalized amplitude of the signal, and w is the signal acquisition time;

[0008] Next, the EEG signal is enhanced; the enhancement method is to increase noise by controlling the noise factor and the signal-to-noise ratio, and to control waveform displacement and waveform stretching.

[0009] Step 2: Creating the training and testing datasets, specifically:

[0010] The SSVEP EEG signals processed in the previous step were used to establish training and test datasets; the training and test datasets were randomly grouped in an 8:2 ratio.

[0011] Step 3: Pulse coding of SSVEP EEG signals, specifically:

[0012] A convolutional pulse coding module, constructed from a convolutional network, performs pulse coding on the processed SSVEP EEG signal using machine learning methods. The network structure of the convolutional pulse coding module includes two convolutional modules and one encoding module. The convolutional modules include convolutional layers, normalization layers, and activation layers; the encoding module uses a pulse output layer. The convolutional pulse coding module outputs high-frequency pulses to pixels with EEG data waveforms, and outputs 0Hz or low-frequency pulses when the EEG data is not at the pixel location, i.e., no pulse or a small number of pulses are output, thereby realizing the pulse coding method.

[0013] Step four, the identification of SSVEP EEG signals, specifically:

[0014] The SSVEP EEG signal recognition uses a spiking neural network to identify and classify the EEG signals. The spiking neural network includes a first residual module, a first pulse connection module, a second residual module, a second pulse connection module, and a pulse fully connected output module arranged in sequence.

[0015] in,

[0016] The first residual module includes a pulse convolutional layer, a pulse firing layer, and a pulse firing layer; the first residual module is connected to the output of the convolutional pulse coding module.

[0017] The first pulse connection module includes a pulse delivery layer and a pulse pooling layer;

[0018] The second residual module includes a pulsed convolutional layer and a pulsed firing layer;

[0019] The second pulse connection module is the pulse delivery layer;

[0020] The pulse fully connected output module includes a pulse fully connected layer and a pulse output layer;

[0021] The pulse data output from step three is processed through the aforementioned spiking neural network and then through the fully connected spiking output module to obtain the classification of SSVEP EEG signals.

[0022] Step 5, follow-up processing, specifically:

[0023] The pulse signals of each category are decoded to obtain the corresponding EEG signals, and the results are displayed; thus, the classification of EEG signals is completed.

[0024] Furthermore, in the step one of acquiring and preprocessing SSVEP EEG signals, the acquisition time is T seconds, and the SSVEP EEG signal is valid EEG data acquired within T seconds.

[0025] Furthermore, in step four, the SSVEP EEG signal recognition step involves building a spiking neural network using the PyTorch and Spac libraries. This spiking neural network is trained using the STAC learning algorithm for at least one epoch. One epoch is the result of training once using all the samples in the training set.

[0026] The beneficial effects of this invention are as follows:

[0027] The method of this invention adopts a combination of convolutional neural networks and spiking neural networks to design the network structure of the recognition network. It uses convolutional neural networks to complete pulse encoding and uses brain-like spiking neural networks to solve the recognition of weak EEG signals, which can reduce power consumption. The pulse firing method reduces the computation time and also solves the problem of training generalization ability with small samples. The network training results have high robustness in recognizing EEG signals. Attached Figure Description

[0028] Figure 1 SSVEP EEG signal recognition flowchart;

[0029] Figure 2 This is a steady-state visual evoked potential map;

[0030] Figure 3 This is the original signal EEG;

[0031] Figure 4 This is a diagram of the convolutional pulse coding network structure;

[0032] Figure 5 This is a diagram of a network structure combining convolutional and spiking neural networks.

[0033] Figure 6 The loss curve refers to the difference between the model's prediction and the actual result as the number of training iterations increases in machine learning.

[0034] Figure 7 The spiking neural network EEG classification and recognition curve is shown. The val loss curve refers to the error of the model on the validation set, indicating the magnitude of the prediction error of the model for unknown data during the training process.

[0035] Figure 8 The accuracy curve refers to the change in the model's accuracy on the test set, which is the proportion of correct data in the prediction results out of the total data.

[0036] Figure 9 A confusion map for EEG classification using a spiking neural network;

[0037] Figure 10 This is a schematic diagram of the firing of neuronal pulses. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific implementation method one:

[0040] This embodiment presents an SSVEP EEG signal recognition method based on convolutional and spiking neural networks, such as... Figure 1 As shown, the method is implemented through the following steps:

[0041] Step 1, Acquisition and Preprocessing of SSVEP EEG Signals, specifically:

[0042] like Figure 2 The steady-state visual evoked point diagram shown indicates that the SSVEP signal is generated by low-frequency visual stimulation of 8Hz–30Hz, followed by EEG signal acquisition via a brain-computer interface device, and then based on the data collected by the BCI device. n(n = 0, 1, 2, ..., N) channel EEG signals, and divide them into several data segments, such as Figure 3 As shown; where, 2 n The acquisition channel acquires the number of input channels c, and its data structure is h×w×c; the segmentation operation is specifically based on time t. i Starting with i = 0, 1, 2, ..., n, the EEG data interval is t seconds, and the data length ends with N data points;

[0043] Then, each data segment is stored as an image, i.e., I w×h h is the normalized amplitude of the signal, and w is the signal acquisition time;

[0044] Next, the EEG signal is enhanced; the enhancement method is to increase noise by controlling the noise factor and the signal-to-noise ratio, and to control waveform displacement and waveform stretching.

[0045] Step 2: Creating the training and testing datasets, specifically:

[0046] The SSVEP EEG signals processed in the previous step were used to establish training and test datasets; the training and test datasets were randomly grouped in an 8:2 ratio.

[0047] Step 3: Pulse coding of SSVEP EEG signals, specifically:

[0048] like Figure 4 As shown, the convolutional pulse coding module 6, constructed using a convolutional network, performs pulse coding on the processed SSVEP EEG signal using a machine learning method. The network structure of the convolutional pulse coding module 6 includes two convolutional modules and one coding module. The convolutional modules include a convolutional layer 12, a normalization layer 13, and an activation layer 14. The coding module uses a pulse output layer 15. Pulse output is achieved through the coding layer. The convolutional pulse coding module is a pre-trained network constructed using a convolutional neural network, employing a simplified network structure to generate pulse-coded data and complete pulse delivery. Specifically, the convolutional pulse coding module 6 delivers high-frequency pulses to pixels with EEG waveforms, while delivering 0Hz or low-frequency pulses (i.e., no pulse or a small number of pulses) when the EEG data is not at a pixel location, thus achieving pulse coding. The data structure of the SSVEP EEG signal is 80*160*8, the data structure of the convolutional modules is 80*160*8, the data structure of the coding module is 80*160*8, and the data structure of the generated pulse-coded data is 80*160*8.

[0049] Step four, the identification of SSVEP EEG signals, specifically:

[0050] like Figure 5 As shown, the SSVEP EEG signal recognition uses a spiking neural network to identify and classify the EEG signals. The spiking neural network includes a first residual module 1, a first pulse connection module 2, a second residual module 3, a second pulse connection module 4, and a pulse fully connected output module 5 arranged in sequence.

[0051] The data structure of the first residual module 1 is 80*160*16; the data structure of the first pulse connection module 2 is 40*80*32; the data structure of the second residual module 3 is 40*80*32; the data structure of the second pulse connection module 4 is 20*40*8.

[0052] in,

[0053] The first residual module 1 includes a pulse convolutional layer 7, a pulse firing layer 8, and a pulse firing layer 8 arranged sequentially; the first residual module 1 is connected to the output of the convolutional pulse coding module 6; the data structure of the convolutional pulse coding module 6 is 80*160*8;

[0054] The first pulse connection module 2 includes a pulse delivery layer 8 and a pulse pooling layer 9;

[0055] The second residual module 3 includes a pulsed convolutional layer 7 and a pulsed firing layer 8;

[0056] The second pulse connection module 4 is the pulse delivery layer 8;

[0057] The pulse fully connected output module 5 includes a pulse fully connected layer 10 and a pulse output layer 11; the data structure of the pulse fully connected layer 10 is 6400; the data structure of the pulse output layer 11 is 38.

[0058] The pulse data output from step three, after passing through the aforementioned spiking neural network, is then processed by the fully connected spiking output module 5 to obtain the classification of the SSVEP EEG signal; the classification and recognition curve is shown below. Figure 6-8 As shown, the spiking neural network EEG classification confusion diagram is as follows: Figure 9 As shown.

[0059] Step 5, follow-up processing, specifically:

[0060] The pulse signals of each category are decoded to obtain the corresponding EEG signals, and the results are displayed; thus, the classification of EEG signals is completed. Specific Implementation Method Two:

[0062] Unlike Specific Implementation Method 1, in this implementation method of SSVEP EEG signal recognition based on convolutional and spiking neural networks, the acquisition time of the SSVEP EEG signal acquisition and preprocessing step 1 is T seconds, and the SSVEP EEG signal is valid EEG data acquired within T seconds. Specific implementation method three:

[0064] Unlike Specific Implementation Method 2, in this implementation method of SSVEP EEG signal recognition based on convolutional and spiking neural networks, the EEG signal recognition network in step four is constructed using the PyTorch and Spac libraries to build a spiking neural network. The spiking neural network is trained using the STAC learning algorithm for at least one epoch; where one epoch is one training session using all samples in the training set.

[0065] SSVEP EEG signal generation process:

[0066] Evoked brain signals are generated through external stimulation of the brain, such as the auditory, tactile, or visual systems. For SSVEP signals, the stimulation is achieved through the visual system, using stroboscopic stimulation to generate evoked potentials in the brain. Electrodes attached to an EEG cap are used to collect EEG signals. Steady-state visual evoked EEG signals are in the microvolt range and are subject to interference from other signals during stimulation. To accurately obtain the classification results of the EEG signals, the collected EEG signals are preprocessed, including signal amplification and signal filtering. The processed signals are then subjected to signal feature extraction and further classification.

[0067] Characteristics of SSVEP EEG signals:

[0068] Steady-state visual evoked potentials (SSVEPs) are also a type of electroencephalogram (EEG) signal, and they are susceptible to interference from other physiological electrical signals and power frequency signals. When acquiring EEG signals through brain-computer interface devices, the physiological electrical signals that can be acquired include electrooculogram (EOG) signals generated by eye movements and electromyogram (EMG) signals generated by limb movements; power frequency signals exist during the signal acquisition process and are caused by interference from the mains power frequency. Although SSVEP EEG signal acquisition introduces perturbation signals, it also possesses its own signal characteristics:

[0069] (1) The SSVEP EEG signal is weak. Considering the amplitude of the acquired waveform signal, its amplitude is below 10uV. Considering the frequency of the acquired waveform, its effective frequency is between 5 and 50Hz. Due to the active visual stimulation, it is clearly shown in the main frequency and harmonic frequency.

[0070] (2) SSVEP EEG signals are periodic. The steady-state visually induced EEG signals collected by the acquisition device are data within a certain time period. When viewed in the time domain, the waveform appears to be a chaotic signal due to the influence of the disturbance signal. However, when observed in the frequency domain, it has obvious spectral data at the stimulation frequency.

[0071] Analyzing the generation and characteristics of EEG signals under steady-state vision reveals that while they appear to have obvious distinguishable features, significant uncertainties exist during signal acquisition. Interference occurs constantly, greatly impacting the EEG signals, and sometimes the acquired perturbation signals are stronger than the useful signals. Therefore, different methods are needed to distinguish the EEG signals. Methods that directly identify EEG signals without training include Power Spectral Density Analysis (PSDA) and Canonical Correlation Analysis (CCA). Methods requiring training include User-Specified EEG Analysis Algorithm (UD), User-Unspecified EEG Analysis Algorithm (UI), and the deep learning-based SSVEP classification method.

[0072] Introduction to spiking neural networks:

[0073] Spiking Neural Networks (SNNs), as third-generation artificial intelligence networks, mimic the information transmission mechanism of human brain cells and belong to the category of brain-inspired artificial intelligence. Currently widely used deep learning networks employ a fully connected network structure, where information is transmitted continuously from input to intermediate layers and then from intermediate layers to the output layer, ensuring complete information preservation and demonstrating good performance in many recognition and classification tasks. However, from a neuroscience or biological science perspective, they do not fully mimic the transmission mechanism of neurons in the human brain. Spiking Neural Networks are a novel type of neural network designed to address this challenge.

[0074] Unlike other networks, SNNs transmit signals in a pulse manner, meaning discrete signals occurring at a specific point in time, rather than continuous data. The frequency of these discrete values ​​represents biological characteristic signals, and the frequency of these signals affects the membrane potential of each neuron. Once a certain threshold is reached, the neuron fires a pulse, transmitting pulse information to the next neuron and simultaneously setting the membrane potential.

[0075] Spiking neural networks mimic the information transmission mechanism of biological neurons. Each neuron receives a pulse signal as input and outputs a pulse signal. Therefore, the identified data needs to be encoded into pulse signals. Encoding methods include rate coding, time coding, and Poisson coding. The encoded data is then converted into pulses and sent to the neuron. Various equivalent models of neurons exist, such as the HH model, LIF model, Izhikevich model, and SRM model. The pulse firing process is completed through these equivalent models. Figure 10The diagram shown illustrates the firing of neuron pulses. As one of the key technologies for information transmission in spiking neural networks, the quality of a single neuron directly affects the information recognition rate.

[0076] Drawing on traditional deep learning optimization methods and technologies from biological neurons and other fields, numerous spiking neural network learning algorithms have been proposed, such as unsupervised learning algorithms, supervised learning algorithms, reinforcement learning algorithms, and ANNtoSNN algorithms. This invention designs a spiking neural network structure that connects each neuron in a fully connected manner, but after training, it becomes a sparse connection. The strength of the effect of one neuron on the next varies depending on the training parameters, affecting the firing of neuronal spiking.

[0077] The embodiments disclosed herein are preferred embodiments, but are not limited thereto. Those skilled in the art can readily grasp the spirit of the present invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for SSVEP electroencephalogram (EEG) signal recognition based on convolutional and spiking neural networks, the method being implemented through the following steps: Step 1, Acquisition and Preprocessing of SSVEP EEG Signals, specifically: Low-frequency visual stimulation of 8Hz~30Hz, followed by EEG signal acquisition via brain-computer interface device, and then based on the data collected by BCI device 2 n Channel EEG signals, n = 0, 1, 2, ..., N, And it is divided into several data segments; among them, Collected 2 n The EEG signal is used as the channel dimension c in the input tensor of the neural network, and its data structure is h×w×c; the segmentation operation is specifically performed by time... t i To begin, i = 0, 1, 2, ..., k The EEG data are spaced t seconds apart, and the data length ends with L data points. Then, each data segment is stored as an image, i.e. I p×q p is the normalized amplitude of the signal, and q is the signal acquisition time; Next, the EEG signal is enhanced; the enhancement method is to increase noise by controlling the noise factor and the signal-to-noise ratio, and to control waveform displacement and waveform stretching. Step 2: Creating the training and testing datasets, specifically: The SSVEP EEG signals processed in the previous step were used to establish training and test datasets; the training and test datasets were randomly grouped in an 8:2 ratio. Step 3: Pulse coding of SSVEP EEG signals, specifically: The convolutional pulse coding module (6) constructed by the convolutional network performs pulse coding on the processed SSVEP EEG signal using machine learning methods. The network structure of the convolutional pulse coding module (6) includes two convolutional modules and one coding module. The convolutional module includes a convolutional layer (12), a normalization layer (13), and an activation layer (14). The coding module uses a pulse output layer (15). The convolutional pulse coding module (6) outputs high-frequency pulses for pixels with EEG data waveforms, and outputs 0Hz or low-frequency pulses when the EEG data is not at the pixel position, i.e., no pulse or a small number of pulses are output, thereby realizing the pulse coding method. Step four, the identification of SSVEP EEG signals, specifically: The SSVEP EEG signal recognition uses a spiking neural network to identify and classify the EEG signal. The spiking neural network includes a first residual module (1), a first pulse connection module (2), a second residual module (3), a second pulse connection module (4), and a pulse full connection output module (5) arranged in sequence. in, The first residual module (1) includes a pulse convolutional layer (7), a pulse firing layer (8), and a pulse firing layer (8); the first residual module (1) is connected to the output of the convolutional pulse coding module (6); The first pulse connection module (2) includes a pulse delivery layer (8) and a pulse pooling layer (9); The second residual module (3) includes a pulsed convolutional layer (7) and a pulsed firing layer (8); The second pulse connection module (4) is the pulse delivery layer (8); The pulse fully connected output module (5) includes a pulse fully connected layer (10) and a pulse output layer (11). The pulse data output in step three is processed by the above-mentioned spiking neural network and then by the fully connected spiking output module (5) to obtain the classification of SSVEP EEG signals; Step 5, follow-up processing, specifically: The pulse signals of each category are decoded to obtain the corresponding EEG signals, and the results are displayed; thus, the classification of EEG signals is completed.

2. The SSVEP EEG signal recognition method based on convolutional and spiking neural networks according to claim 1, characterized in that: In the step one, the acquisition and preprocessing of SSVEP EEG signals, the acquisition time is T seconds, and the SSVEP EEG signal is the valid EEG data acquired within T seconds.

3. The SSVEP EEG signal recognition method based on convolutional and spiking neural networks according to claim 1 or 2, characterized in that: In step four, the SSVEP EEG signal recognition process involves building a spiking neural network using the PyTorch and Spac libraries. This spiking neural network is trained using the STAC learning algorithm for at least one epoch. One epoch is the number of training cycles using all samples in the training set.

Images