A spatio-temporal joint pulse neural network electroencephalogram signal recognition method and system

Abstract

The application discloses a kind of spatio-temporal joint pulse neural network electroencephalogram signal identification method and system.The application carries out three-stage multi-scale feature extraction to the electroencephalogram signal after pulse coding of pulse neural network, in the feature extraction process, introduce pulse attention mechanism and spatio-temporal pulse enhancement mechanism, to improve the modeling ability of complex spatio-temporal dependence relationship of electroencephalogram signal.Construction is based on the attention calculation mode of pulse signal, guide network in time dimension and channel dimension selectively focus on the pulse response more critical to classification task.The final obtained high-level pulse feature is input into the classification module, and the final identification result of electroencephalogram signal is output by averaging the prediction results of all time steps.The application improves the accuracy, stability and explainability of electroencephalogram signal recognition while maintaining the event-driven and low-power characteristics of pulse neural network.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and neural engineering, and relates to a method and system for electroencephalogram (EEG) signal recognition using a spatiotemporal joint spiking neural network (SNN). Background Technology

[0002] Electroencephalography (EEG), a non-invasive bioelectrical signal recording of brain neuronal population activity, is widely used in tasks such as diagnosis of neuropsychiatric diseases, brain-computer interfaces, emotion recognition, sleep analysis, and motor imagery.

[0003] In recent years, deep learning methods based on Artificial Neural Networks (ANNs) have been widely used for EEG signal recognition. Typical approaches include using convolutional structures to extract local time-frequency features, utilizing recurrent or graph structures to model inter-channel dependencies, and introducing self-attention mechanisms to capture long-range temporal dynamics. While these methods have achieved high classification accuracy on multiple public datasets, they generally suffer from the following drawbacks: reliance on numerous floating-point multiplication operations, resulting in high computational overhead and low energy efficiency; lack of correspondence with biological neural mechanisms; and a lack of interpretability in the model's decision-making process, making it difficult to meet the needs of high-reliability scenarios such as medical treatment.

[0004] To overcome the aforementioned problems, spiking neural networks (SNNs) are considered a more biologically sound and energy-efficient alternative due to their event-driven nature, sparse binary impulse propagation, and computational characteristics that replace multiplication with addition. SNNs are particularly well-suited for processing biological signals such as EEG, which are inherently temporal and sparse.

[0005] However, existing SNN-based EEG recognition methods still have significant technical limitations: (1) The architecture design is not adapted to the characteristics of SNN: Most solutions directly convert the ANN structure (such as convolution, recurrent or graph network) into pulse form without optimizing for the essential characteristics of SNN such as multi-time step dynamics and pulse sparsity. They fail to make full use of the event-driven and sparse pulse characteristics of SNN, making it difficult to effectively model the joint relationship between the spatial correlation and temporal dynamics between channels in EEG signals. (2) Attention mechanisms are not suitable for EEG structures: Although some studies have introduced self-attention mechanisms into SNNs, they are mostly derived from image processing paradigms and cannot simultaneously preserve the multi-channel spatial topology and long-term sequence dynamics of EEG signals. They are also sensitive to noise and have poor robustness. On the other hand, existing pulse attention mechanisms either have high computational complexity or cannot simultaneously preserve the multi-channel structure and temporal characteristics of EEG, and often rely on floating-point multiplication, making them difficult to adapt to neuromorphic hardware.

[0006] (3) Lack of regularization methods adapted to SNN: The regularization methods commonly used in traditional ANN, such as Dropout, not only fail to alleviate overfitting in small samples, but also lead to training instability and performance degradation because they disrupt the continuity of the timing of pulse firing in SNN. Moreover, in sample scenarios with a limited number of subjects, traditional regularization methods (such as Dropout) will disrupt the timing pulse firing mechanism of SNN, resulting in training instability and insufficient generalization ability; In summary, current technologies lack a universal recognition framework specifically designed for SNN architectures that can efficiently integrate EEG spatiotemporal features and possess strong robustness and generalization capabilities. Therefore, there is an urgent need to propose a novel spiking neural network structure that can effectively address the shortcomings of existing technologies in feature extraction, attention modeling, and regularization while maintaining the biological rationality and low power consumption advantages of SNNs, thereby improving the accuracy, stability, and interpretability of EEG signal recognition. Summary of the Invention

[0007] The purpose of this invention is to address the problems of insufficient spatiotemporal feature modeling ability, low information utilization efficiency of spiking neural networks, and limited generalization performance in existing EEG signal recognition methods, and to propose a spatiotemporal joint spiking neural network EEG signal recognition method and system.

[0008] A spatiotemporal joint spiking neural network method for EEG signal recognition specifically includes the following steps: Step 1: Organize the continuous EEG signals of the subject into a multidimensional data representation that includes both channel spatial information and time sequence information, and pulse-encode the multidimensional data representation to generate a binary pulse sequence covering multiple pulse time steps; Step 2: Input the binary pulse sequence into a multi-scale spiking neural network feature extraction architecture for spatiotemporal feature extraction. The specific feature extraction mechanism includes: extending the learnable latent encoding vector in the time dimension to construct a unified global pulse query benchmark, and performing attention interaction operations based on the query benchmark and the pulse features at the corresponding time step to generate latent encoded pulse attention output; accumulating the pulse query signal in the time dimension to extract temporal evolution features, and accumulating the pulse value signal in the channel dimension to extract channel correlation features; combining the two based on element-wise pulse operations to generate EEG self-attention feature output; simultaneously performing joint operations on the temporal continuity of the pulse sequence and the spatial topological relationship of the multi-channel EEG signals to generate a spatiotemporally enhanced pulse representation; and obtaining multi-scale high-level spatiotemporal pulse features through iterative execution and downsampling processing at different stages using the above mechanism. Step 3: After completing the multi-scale feature extraction, the obtained high-level spatiotemporal pulse features are input into the classification module. The classification module aggregates the features of each time step through global average pooling, obtains the category prediction results corresponding to each time step through fully connected mapping, and outputs the final recognition and classification results of the EEG signal by averaging the prediction results of all time steps.

[0009] Furthermore, in step one, before organizing the subject's EEG signal into a multidimensional data representation, the EEG signal is preprocessed. The preprocessing operation includes filtering out power frequency noise, low-frequency drift, and artifact interference.

[0010] Furthermore, the pulse encoding process in step one is as follows: by utilizing the membrane potential accumulation and firing decay mechanism of spiking neurons in the spiking neural network, the input continuous EEG signal is converted into a binary sequence over multiple pulse time steps.

[0011] Furthermore, in step two, the multi-scale spiking neural network feature extraction architecture includes three stages of downsampling and feature extraction: the first stage sequentially performs spiking attention calculation based on latent encoding vectors and spatiotemporal spiking joint enhancement calculation; the second stage sequentially performs spiking EEG self-attention calculation, spatiotemporal spiking joint enhancement calculation, spiking attention calculation based on latent encoding vectors, and spatiotemporal spiking joint enhancement calculation again; the third stage continuously performs three sets of standard spiking self-attention calculation and spatiotemporal spiking joint enhancement calculation.

[0012] Furthermore, the method also includes a step of visualizing EEG features: statistically analyzing the pulse firing intensity and temporal distribution and channel response difference data induced by different types of EEG signals within the feature extraction architecture; generating a pulse response distribution map containing spatial and temporal response features based on the difference data, and outputting the distribution map to a display device.

[0013] On the other hand, the present invention also proposes an electronic device for implementing the method, the device comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described above. Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described above.

[0014] Compared with the prior art, the present invention has the following significant advantages: (1) Improve the fidelity of EEG signal pulse representation: The present invention adopts a coding method based on multi-time step spiking neurons to convert continuous EEG signals into pulse sequences with a clear time structure. Compared with the traditional single-step threshold coding method, it can more fully preserve the relative amplitude relationship and dynamic change characteristics of EEG signals and effectively reduce information loss in the coding process.

[0015] (2) Achieving spatiotemporal joint modeling that conforms to the topological characteristics of EEG: In view of the high coupling between the multi-channel spatial topology and temporal dynamics of EEG signals, this invention introduces a pulse attention mechanism designed for EEG structure during the feature extraction process. By avoiding the simple flattening of the channel and time dimensions, the risk of structural information confusion is reduced, which is conducive to maintaining the original spatiotemporal organization relationship of EEG signals.

[0016] Meanwhile, by decoupling the channel association and temporal evolution through modeling, and by constructing the attention response using element-wise pulse operations, the computational complexity is kept at a low level while effectively characterizing global dependencies, making it suitable for EEG data processing scenarios with high channels and high temporal resolution.

[0017] (3) Enhance the generalization ability of the model under limited sample conditions: By introducing a latent encoding mechanism, this invention uses learnable latent representations to replace the complete dependence on the original input in the attention modeling process, thereby reducing the sensitivity of the model to the distribution of training samples while maintaining the dynamic characteristics of pulse time, and effectively alleviating the problem of overfitting of spiking neural networks under small EEG sample conditions.

[0018] (4) Enhance the comprehensive expression ability of spiking neural networks for spatiotemporal features: By setting up a spatiotemporal pulse enhancement structure, this invention simultaneously reconstructs the dynamic correlation between adjacent time steps and the spatial topological relationship of multi-channel EEG signals during feature extraction, making up for the shortcomings of traditional spiking neural networks in terms of temporal continuity and spatial structure modeling, thereby improving the ability to discriminate complex EEG patterns.

[0019] (5) End-to-end multi-scale spiking network architecture that balances recognition accuracy and energy efficiency: This invention constructs a multi-stage, multi-scale spiking neural network structure. By compressing time and channel dimensions layer by layer and expanding feature representation, it achieves hierarchical feature abstraction from local to global, improving recognition accuracy while maintaining the inherent advantages of spiking neural networks in event-driven and low-power consumption. The overall architecture is entirely based on spiking neural networks, has high biological rationality and interpretability, and is suitable for deployment on neuromorphic hardware platforms.

[0020] Furthermore, the method of the present invention has good versatility and can achieve better recognition results than existing technologies in various EEG application scenarios with different acquisition devices and different subject groups, verifying its effectiveness and adaptability in multi-task and multi-practical application scenarios. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the electroencephalogram (EEG) recognition method based on a spiking neural network in this embodiment. Figure 2 This is a flowchart illustrating the feature extraction process for potential coded pulse attention blocks in the embodiment. Figure 3 This is a flowchart illustrating the feature extraction process of spatiotemporal pulse blocks in the embodiment. Figure 4 This is a flowchart of feature extraction from pulse EEG self-attention blocks in the embodiment; Figure 5 This is a flowchart illustrating the feature extraction process of a standard pulse self-attention block in the embodiment. Figure 6 The example uses the LIF model for pulse coding and utilizes the saliency map calculated from the PRED+CT dataset using surrogate gradients. Figure 7 In this example, the LIF model is used for pulse coding to visualize the pulsed attention of different categories of subjects in the HUSM dataset. Detailed Implementation

[0022] The execution entity in this embodiment can be a computer device, which includes a processor and a memory. The memory stores computer program instructions for implementing the method of the present invention; when the instructions are executed by the processor, the device is able to complete the EEG data processing and recognition method described in this embodiment of the present invention.

[0023] Specifically, the processor is used to perform the following operations: acquire and preprocess EEG signals; construct a recognition model including a pulse coding module, a three-stage multi-scale SNN feature extraction backbone, and a classification output module; train and test the model using a surrogate gradient algorithm; and use the trained model to recognize the input EEG signals.

[0024] The memory may include random access memory (RAM) or non-volatile memory (such as disk storage) for storing programs and model parameters. The processor may be a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other processing unit capable of executing program instructions. The steps of this invention can be implemented through hardware circuits, software instructions, or a combination of hardware and software.

[0025] In some embodiments, the computer device may also be a processing platform based on neuromorphic hardware (such as Loihi or TrueNorth), or a computing device such as a desktop computer, laptop computer, workstation, or server capable of executing deep learning models.

[0026] like Figure 1 As shown, this invention provides a method for recognizing electroencephalogram (EEG) signals using a spatiotemporal joint spiking neural network, specifically including the following steps: Step 1: Organize the subject's EEG signals into a multidimensional data representation that simultaneously includes channel spatial information and time-series information, and perform pulse encoding on the EEG signals to obtain binary sequences at multiple pulse time steps; specifically as follows: (1) Acquire and preprocess EEG signals: EEG signals were acquired using a conventional multichannel EEG acquisition system. The acquisition process employed an electrode placement method conforming to the international 10–20 or its extended system, placing multiple electrodes on the scalp surface to record electrical activity in different brain regions. During acquisition, the EEG device discretized the continuous voltage signals at a fixed sampling frequency, forming multichannel time-series data. After acquisition, the recorded data was represented as a multichannel time-series matrix, used as input for subsequent pulse coding and feature extraction.

[0027] For different EEG data sources, the number of channels, sampling rate, or electrode configuration may vary, and the preprocessing method of this invention can adapt to these differences. First, invalid channels are discarded, and a 50Hz notch filter is used to remove power frequency interference, along with a 0.1Hz high-pass filter and a 70Hz low-pass filter to remove baseline drift and high-frequency noise. An average reference is used to set the potential benchmark and eliminate reference electrode bias. Subsequently, an independent component analysis (ICA) algorithm (preferably FastICA) is used to separate the signal source, identify, and remove artifact components related to eye movement (EOG). Finally, the first and last 30 seconds of data from each record are truncated to eliminate boundary effects, z-score standardization is performed on the remaining data, and a sliding window method is used for segmentation to expand the sample size.

[0028] (2) Data cross-validation partitioning: To improve the robustness of model training and reduce sample partitioning bias, an N-fold cross-validation strategy is used to partition the data on each dataset.

[0029] The specific partitioning process is as follows: First, the entire dataset is divided into N mutually exclusive subsets (i.e., "folds") of equal size. In each iteration, one subset is selected as the test set, while the remaining N-1 subsets are combined as the training set. This process is repeated N times to generate N different combinations of training and test sets. By averaging or statistically analyzing the results of N independent tests, a more stable and reliable evaluation of the model's generalization ability can be obtained compared to a single random partition, ensuring that the model's effectiveness is not affected by the selection of specific samples.

[0030] (3) Pulse coding: After cross-validation partitioning, the data is input into the pulse coding module of the spiking neural network to convert the continuous waveform into a pulse sequence with a time structure.

[0031] The preprocessed electroencephalogram (EEG) signals were organized into a multidimensional floating-point matrix and then pulsed encoded. This input first underwent feature compression and normalization to reduce data redundancy and improve training stability. Subsequently, a spiking neuron model (in this implementation, the leaky integrate-and-fire (LIF) model) was used... T S A pulse time step (in this embodiment, a time step is used) T S Convert it into a binary pulse sequence within 4).

[0032] The mathematical model of LIF is formally represented as follows: ; in, H(⋅) Let U[t] be the Heaviside function (unit step function), X[t] be the neuron membrane potential at time t, X[t] be the input signal, and S[t] be the binary pulse output at time t, respectively. In this embodiment, the membrane potential decay coefficient of the LIF neuron is... (Leakage coefficient) is set to 0.5, pulse delivery threshold. Set to 1.0. After encoding, the original signal is mapped to a shape of 4 (pulse time steps). T S ) × Feature Dimension F 1×56 (channel dimension) C 1 ) × 56 (EEG time dimension) T 1 The binary impulse tensor of ). Wherein, the feature dimension F 1 Based on the balance between computational resource requirements and recognition accuracy, two preferred model configuration schemes are provided: (1) When using standard parameter configuration, the feature dimension F 1 The value is 128; (2) When using lightweight parameter configuration, the feature dimension F 1 The value is 64.

[0033] This encoding process preserves the original spatial topology (channel arrangement) of the EEG signal while transforming continuous amplitude information into a sparse, event-driven temporal pulse representation, thereby preserving the dynamic change characteristics of the signal and providing a suitable input format for the efficient and low-power processing of subsequent spiking neural networks.

[0034] Step 2: Employ a three-stage multi-scale SNN feature extraction architecture to extract multi-scale spatiotemporal pulse features through joint temporal and spatial modeling. The first stage uses a latent coding spiking attention (LTSA) block + a spatiotemporal spiking STSB block; the second stage uses a spiking EEG self-attention (SESA) block + a spatiotemporal spiking STSB block + a latent coding spiking attention (LTSA) block + a spatiotemporal spiking STSB block; and the third stage uses three sets of consecutive standard spiking self-attention (SSA) blocks + spatiotemporal spiking STSB blocks. The network depth is gradually increased by stacking multiple feature extraction units.

[0035] When executing the next stage, the input pulse features (the output features of the previous stage) are downsampled in both the channel and time dimensions, reducing the size to half that of the previous stage; simultaneously, the feature dimensions are expanded, increasing them to four times that of the previous stage.

[0036] The EEG pulse coding results obtained through pulse coding are used as input and fed into a three-stage multi-scale spiking neural network (SNN) feature extraction architecture. Multi-scale spatiotemporal pulse features are extracted through joint temporal and spatial modeling to achieve hierarchical spatiotemporal feature abstraction from local to global perspectives. This multi-scale feature extraction architecture selects different types of pulse attention blocks at different stages, corresponding to the progressive abstraction and enhancement of multi-scale features.

[0037] At the starting position, a latent encoded pulse attention block is used to extract features from the pulse features of the input at the current stage, so as to reduce the direct dependence of the model on the distribution of training samples and improve the generalization ability in small sample EEG data scenarios.

[0038] Assume the first The pulse characteristics of the latent encoded pulse attention block input at each stage are X. LTSA Its dimension is the pulse time step. × Feature Dimension ×Channel Time Aggregation Dimension Define a set of learnable potential tokens L, with an initial dimension of . The latent token is used to generate the query signal (Q) in the attention mechanism, while the key signal (K) and value signal (V) are generated directly from the input impulse features of the current stage.

[0039] like Figure 2 As shown, the specific steps are as follows: First, the potential token L is normalized to generate the query signal Q; second, the input impulse feature X is... LTSA After convolution and normalization, feature mapping and impulse transformation are performed to generate keys K and values ​​V. To maintain the temporal dynamic consistency of the spiking neural network, the generated query signal Q is broadcast to all impulse time steps (e.g., the same Q is copied in four time steps), ensuring that each time step shares a unified query representation to maintain the temporal dynamics of the SNN. The computation process can be represented as follows: ; Where BN represents normalization, W K and W V The eigenmap matrices of K and V are respectively, and SN is the eigenmap matrix of V. K 、 SN V This represents the spiking neuron layer used for pulse conversion.

[0040] Subsequently, based on the broadcast Q and the pulsed K and V, pulse attention calculation is performed to obtain the LTSA output: ; Finally, the output of LTSA is restored to be the same as the input X after pulse one-dimensional and two-dimensional convolution mapping. LTSA Same shape.

[0041] In this calculation process, by ensuring that K and V involved in the operation are binary impulse matrices, the multiplication operation is equivalent to an addition operation, thus avoiding the introduction of complex floating-point multiplication. This mechanism effectively alleviates the problem of overfitting with small samples by weakening the model's direct dependence on training samples; at the same time, it does not require the introduction of destructive regularization operations such as Dropout, and fully preserves the dynamic correlation of the SNN impulse sequence across multiple time steps, improving training stability and classification robustness.

[0042] After each pulse attention block, its output is fed into a spatiotemporal pulse STSB block to enhance the spatiotemporal representation of pulse features. For example... Figure 3 As shown, the spatiotemporal pulse STSB block includes a time branch TSB and a spatial branch SSB.

[0043] (1) Time Branch (TSB) The time branch is used to reconstruct the dynamic correlation between adjacent time steps lost due to pulse binarization. In this embodiment, the input pulse tensor is assumed to be... Containing n pulse time steps (e.g., n=4), the input is first divided into n-1 overlapping adjacent time blocks along the time dimension. Each block contains two consecutive time steps, i.e.: ; Among them B i,i+1 Indicates that by the first i With the i Adjacent blocks consisting of +1 time steps.

[0044] Subsequently, feature enhancement operations (e.g., through learnable impulse-driven transformations) are performed on each adjacent block to obtain the enhanced block representation. Crucially, to reconstruct temporal continuity, adjacent blocks with odd indices (e.g., [1,2], [3,4], ...) and adjacent blocks with even indices (e.g., [2,3], [4,5], ...) are concatenated along the time dimension, and the arithmetic mean is calculated at overlapping time steps (e.g., steps 2, 3, and 4). This process can be formally represented as: ; Where Concat represents the dimension T S The concatenation operation, Average, represents the element-wise averaging across overlapping time steps. This operation effectively bridges the gaps between blocks, restores dynamic dependencies across time steps, and the averaging operation can be implemented by adjusting the floating-point exponent, avoiding multiplication operations.

[0045] (2) Spatial Branch (SSB) The spatial branch is used to extract multi-channel spatial features from EEG signals. This branch processes the input through a pulse-driven spatial feature extraction unit and introduces feature reuse mechanisms (such as residual connections) to enhance the modeling ability of inter-channel relationships. Specifically, the input is subjected to multi-level pulsed two-dimensional convolution with residual connections and normalization to form an enhanced spatial representation.

[0046] (3) Spatiotemporal weighted fusion Finally, the time branch output and the spatial branch output are weighted and fused. In this embodiment, the weight coefficient of the time branch is set to 0.5. This scaling is achieved by adjusting the exponent of the floating-point number, avoiding the introduction of multiplication operations. The final STSB output is: ; In the second stage, the pulse signal is fed into the Pulsed EEG Self-Attention (SESA) block, thereby jointly modeling the interchannel spatial correlation and temporal dynamics of the EEG signal in the spiking neural network. In this embodiment, the input to the SESA module is the feature tensor X from the previous stage. SESA Its dimensions are (Pulse time step × feature dimension × channel dimension × time dimension).

[0047] First, consider the input feature tensor X. SESA Feature remapping is performed to preserve its spatiotemporal features during dimensionality reduction, resulting in a shape of... The intermediate representation is X'. Subsequently, channel dimension restoration is performed on the remapped features: through a one-dimensional feature compression operation (projecting along the merged feature dimension), it is transformed into... The compressed pulse tensor Y is used to facilitate subsequent attention calculations: ; ; Where SN(⋅) is the spiking neuron layer, and W is the feature compression matrix.

[0048] Next, as Figure 4 As shown, after performing three sets of one-dimensional convolutions and normalization on tensor Y, impulse mapping is performed to generate query (Q), key (K), and value (V). To separate channel and temporal attention, Q is summed in the time dimension to obtain a shape of... The channel feature representation Q' is obtained by summing V along the channel dimension, resulting in a shape of... The temporal characteristics are represented by V'. The impulse attention calculation, which involves the summation result in element-wise multiply computation, can be expressed as: ; ; ; Among them W Q W K W V The output transformation matrices for Q, K, and V are SN, respectively. Q SN K SN V This corresponds to the spiking neuron layer. Q' and V' represent the summation results in the time and channel dimensions, respectively. This represents element-wise multiplication. The computational complexity of this process is linear O(C+T), which is significantly better than the traditional self-attention method's O(TC).

[0049] Finally, through the reverse dimension transformation operation, Expand back And restore the output to be the same as the input. The structure ensures that SESA modules can be stacked in the network.

[0050] The third stage employs a standard Spiking Self Attention (SSA) block. The core of SSA lies in using pulse signals (binary sequences containing only 0s and 1s) as queries, keys, and values ​​for computation, avoiding the floating-point multiplication operations found in traditional self-attention. Its computation process is implemented through logical AND operations and addition. The membrane potential dynamically follows the Leaky Integrate-and-Fire model, firing a pulse when the membrane potential reaches a threshold. Since pulse-based computation naturally generates a non-negative attention map, SSA does not require Softmax normalization, thus reducing computational overhead. This embodiment uses... Figure 5 The standard impulsive self-attention mechanism (SSA) shown performs three sets of convolutions and normalizations on the input feature tensor, and then uses impulsive multiply-accumulate operations to calculate the correlation between three weight matrices: query (Q), key (K), and value (V). Step 3: The classification module performs category prediction. The pulse feature tensor output by the three-stage multi-scale SNN feature extraction architecture described above is fed into the category classification module for prediction. In this embodiment, the shape of the final output pulse feature tensor is... (Pulse time step × feature dimension × channel dimension × time dimension).

[0051] First, by performing global average pooling on both the channel dimension and the temporal sampling dimension, the features are compressed into a shape of... The temporal feature vector retains the high-dimensional semantic information of each pulse time step.

[0052] Subsequently, the temporal feature vector is subjected to a category mapping transformation, projecting it onto the category semantic space to obtain a shape of The predicted original classification output (where n (Number of categories for EEG signals).

[0053] Finally, to fully utilize the temporal accumulation effect of spiking neural networks, the following is performed: T S The prediction results for each pulse time step are arithmetically averaged to obtain the final result. n The dimensional classification probability distribution serves as the model's classification result for the input EEG signal.

[0054] This classification mechanism effectively improves discrimination stability and noise resistance by integrating decision information from multiple time steps, while maintaining the integrity of the SNN event-driven architecture.

[0055] To further reveal the internal dynamic characteristics of spiking neural networks when processing different types of EEG signals, an interpretability analysis of the model's decision-making process was conducted using visualization. By statistically analyzing and displaying the spiking activity and characteristic responses after EEG signals are input into the network, the differences in the network's performance of different types of EEG signals are characterized from aspects such as neural dynamics, attention distribution, and channel contribution, providing support for understanding model behavior and subsequent applications. Specifically: By statistically analyzing the average pulse firing rate of spiking neurons in each layer of the network within a time window, the intensity of pulse activity induced by different types of EEG signals is compared to reflect the overall differences in pulse response exhibited by different categories during the feature extraction stage. Simultaneously, the binary pulse feature matrix output by the pulse attention module is visualized to observe the attention distribution of the model in the time or channel dimensions, thereby revealing the differences in attention patterns of different types of EEG signals at temporal locations or local regions.

[0056] In addition, the contribution of each input EEG channel to the classification results was evaluated by using saliency maps, and the channel importance distribution corresponding to each category was obtained, thus reflecting the discrimination criteria of the model in the spatial dimension.

[0057] Meanwhile, to achieve end-to-end training, this embodiment employs a direct training strategy to optimize the spiking neural network. Since the firing function of a spiking neuron is a step function, which is non-differentiable during backpropagation, this embodiment introduces a surrogate gradient function to replace the true gradient, enabling the network weights to be effectively updated through error backpropagation.

[0058] Specifically, during forward propagation, the network outputs a class score vector at each pulse time step; during training, it first performs training on all pulse time steps (e.g., ...). T S The output of (=4) is arithmetically averaged over time to obtain the final average category score; then, the average score is converted into the predicted probability of each category through a normalized exponential function (i.e., the softmax function).

[0059] The loss function is in the form of cross-entropy, which is the negative log-likelihood between the true label (using one-hot encoding) and the predicted probability. This loss function is used to drive the backpropagation process.

[0060] In backpropagation, the gradient of the spiking neuron is replaced with a continuously differentiable surrogate function. In this embodiment, the surrogate gradient is used with a firing threshold. V th (This implementation plan is preferred) V th A smooth function centered at (=1) has a shape determined by the hyperparameters. α(Preferred selection) α= 2) Control: ; in u This refers to the neuronal membrane potential. S This function outputs impulses. It exhibits a high gradient response near the threshold, while the gradient decays rapidly away from the threshold, thus ensuring training stability while preserving the sparsity of impulse output.

[0061] Through the above mechanism, this invention achieves effective gradient backpropagation and parameter updates while maintaining forward propagation as a pure binary impulse operation, thereby completing end-to-end training of the SNN model. In practical use, the model can be trained on each fold training set according to the data partitioning described in step one, and validated on the corresponding test set. Finally, by aggregating the test results of each fold experiment, a robustness assessment of the generalization ability of the model of this invention can be obtained. However, the partitioning method does not affect the specific implementation of the training method described in this invention.

[0062] To verify the applicability of the method of the present invention in different types of EEG recognition tasks, this embodiment selects several publicly available EEG datasets for testing. The selected datasets cover typical application scenarios such as mental illness discrimination, human emotion recognition, and motor imagery classification, and can be used to verify the robustness of the method of the present invention under different channel configurations, different sampling rates, and different task types.

[0063] The selected dataset is as follows: Dataset 1: HUSM Dataset (Mental Illness Discrimination) This dataset includes 34 patients with depression and 30 healthy controls. EEG signals were acquired using 19 electrodes according to the international 10–20 system at a sampling frequency of 256 Hz. In this embodiment, continuous 5-minute EEG recordings were performed with the subjects at rest with their eyes closed.

[0064] Dataset 2: MODMA Dataset (Mental Illness Differentiation) The study included 24 patients with depression and 29 healthy controls. EEG data were acquired using a 128-channel HydroCel geodesic sensor array at a sampling frequency of 250 Hz for 5 minutes of closed-eye resting-state EEG recordings, with subjects remaining awake and without physical movement.

[0065] Dataset 3: DEAP dataset (Human Emotion Recognition) The study included 32 participants, each of whom watched 40 one-minute music videos and subjectively rated four affective dimensions—valence, arousal, dominance, and liking—on a scale of 1–9. EEG signals were acquired using a 32-channel system with a sampling frequency of 512 Hz. This dataset was used for a binary valence classification task, with a score of 5 as the threshold for classifying participants into high / low valence categories.

[0066] Dataset 4: PhysioNet dataset (motion imagination classification) The dataset contains EEG recordings of motor imagery from 109 participants. The tasks included four categories of movements: left hand clenching, right hand clenching, both hands clenching, and foot movements. Each task lasted approximately 4 seconds. EEG signals were acquired through 64 effective channels at a sampling frequency of 160Hz. This embodiment uses the complete recordings of the motor imagery portion of this dataset for the four-class classification task.

[0067] Dataset 5: PRED+CT Dataset (Mental Illness Differentiation) The study included 121 participants, who were grouped according to their Beck Depression Inventory (BDI) scores: those with a BDI < 7 were assigned to the healthy control group, and those with a BDI ≥ 13 were assigned to the depression patient group. Ultimately, 44 depression patients and 74 healthy controls were included. EEG signals were acquired during a 500-second resting state with eyes closed, using 64 scalp electrodes and 2 electrooculography (HEOG, VEOG) channels. The electrode arrangement followed the 10–20 international standard, and the sampling frequency was 500 Hz.

[0068] The model's performance is evaluated using commonly used quantitative metrics in the field, primarily including: (1) Accuracy: It is used to measure the proportion of the number of correctly classified samples in the total number of test samples and is a basic indicator reflecting the overall recognition ability.

[0069] (2) Precision: This measures the proportion of samples that are identified as belonging to a certain category by the model but actually belong to that category. It can reflect the false detection situation of the model.

[0070] (3) Sensitivity (also known as Recall): It is used to measure the proportion of samples that actually belong to a certain category that are successfully identified by the model, reflecting the missed detection situation.

[0071] (4) F1 score: The harmonic mean of precision and sensitivity, used to comprehensively reflect classification performance when data classes are imbalanced.

[0072] In the above indicators, the statistics for true positive, false negative, false positive and true negative are all calculated according to the conventional definitions in the classification task.

[0073] Furthermore, to evaluate the operational efficiency of the model during actual deployment, this invention introduces an energy consumption metric. The energy consumption of an Artificial Neural Network (ANN) is estimated by the total number of floating-point multiply-accumulate (MAC) operations performed during the statistical inference phase, assuming that each MAC operation has a fixed energy consumption, thus obtaining an approximate overall energy consumption of the network. The energy consumption of a Spiking Neural Network (SNN) employs an event-driven energy consumption estimation method, estimating it not based on continuous floating-point operations, but rather on the number of synaptic operations triggered by pulse firing. These metrics allow for a comprehensive evaluation of the classification performance, recognition stability, and energy efficiency of this invention.

[0074] To ensure the objectivity and comparability of the evaluation, all comparison methods were tested on the uniformly preprocessed EEG data described in Step 1, using the same cross-validation strategy. All models were trained from random initialization without using any pre-trained weights. For each method being compared, the optimal hyperparameter configuration reported in its publicly available documentation was used for reproduction to ensure the representativeness of the performance evaluation. Detailed comparison results for each dataset are listed in Tables 1 through 5, with evaluation metrics expressed as mean ± standard deviation.

[0075] Table 1. Validation and comparison with existing methods on the HUSM dataset. Table 2. Validation and comparison with existing methods on the MODMA dataset. Table 3. Validation and comparison with existing methods on the DEAP dataset. Table 4. Validation and comparison with existing methods on the PhysioNet MI dataset. Table 5. Validation and comparison with existing methods on the PRED+CT dataset. As can be seen from the experimental data in Tables 1 to 5, the EEG recognition method proposed in this invention significantly outperforms mainstream methods in terms of accuracy, precision, sensitivity, and F1 score on five publicly available datasets, verifying the universality and effectiveness of the architecture in multiple tasks and scenarios. Specifically, in complex dynamic tasks such as DEAP and PhysioNet, the simultaneous improvement in precision and recall effectively suppresses the false negative problem, demonstrating strong robustness. In resting-state disease discrimination tasks such as HUSM, MODMA, and PRED+CT, the outstanding advantages in accuracy and F1 score prove that the model can accurately extract weak pathological features from high-noise EEG, demonstrating high-precision recognition and fine feature analysis capabilities.

[0076] Further comparison with various existing SNN methods used in the experiment reveals that the performance differences mainly stem from variations in temporal modeling strategies and information preservation capabilities. Taking MI_SNN as an example, this method directly maps temporal sampling points in EEG segments to the time step input of the SNN. With long time steps, spiking neurons are prone to significant information loss and gradient decay during multiple integrations and firings, leading to a decline in overall recognition performance. BISNN alleviates the training difficulties caused by long time steps to some extent by dividing EEG segments into multiple independent time windows, each serving as the time step input for the SNN. However, this method still suffers from information compression within windows and disruption of temporal continuity between adjacent windows, resulting in a significant performance gap compared to the method of this invention. In contrast, methods such as Spikformer, QKFormer, and SpikingResformer, as general high-performance models in the SNN field, primarily rely on deep network structures to reduce pulse information loss. However, they are not specifically designed for the inherent spatiotemporal coupling characteristics of EEG signals, making it difficult to fully leverage their modeling advantages in EEG tasks. Among them, Spikformer achieved a relatively limited performance improvement under low energy consumption conditions, while SpikingResformer achieved some accuracy improvement by increasing network complexity, but its performance was still lower than that of the method of this invention, and it was accompanied by higher energy consumption.

[0077] Furthermore, regarding energy consumption analysis, although SNNs theoretically possess low power consumption advantages, in practical applications, the information loss caused by binary impulse representation often limits their energy efficiency while maintaining high accuracy. This invention, through a rationally designed network structure, enhances feature representation capabilities while preserving the event-driven characteristics of spiking neural networks, and on this basis provides standard parameters (encoded feature dimensions). F 1 =128) and lightweight parameters ( F 1=64) Two configuration options are available to meet the different performance and energy consumption requirements of different application scenarios.

[0078] Experimental results show that, regardless of the configuration, this invention fully maintains the event-driven characteristics of spiking neural networks. The standard configuration, while pursuing ultimate recognition accuracy, maintains overall energy consumption on par with existing SNN methods, far lower than ANN models of similar depth. The lightweight configuration, with half the feature dimensions, further optimizes energy efficiency by significantly reducing computational overhead and the total number of pulses fired, while still achieving classification accuracy superior to most existing technologies. As shown in Table 1, this invention achieves superior classification performance at comparable or lower energy consumption levels under both parameter configurations, demonstrating its more prominent and flexible comprehensive technical advantages in practical deployment environments (especially in computationally limited embedded systems).

[0079] like Figure 6 As shown, this figure illustrates the significance map calculated from the PRED+CT dataset using the LIF model for pulsed encoding and surrogate gradients, employing the method described in this embodiment. Darker colors indicate higher correlation. Figure (a) shows the average significance of the healthy control group; (b) shows the average significance of the patients. During the classification decision process for the PRED+CT dataset, electrode channels FZ, FPZ, FP1, FP2, CZ, C3, C4, FT7, FT8, F7, and F8 exhibit significantly higher feature contributions. These channels constitute the key spatial feature mapping region for the method of this invention in determining mental illness.

[0080] like Figure 7As shown, this embodiment uses the LIF model for pulse encoding to visualize the pulsed attention of different categories of subjects in the HUSM dataset. The first three rows correspond to the binary pulse firing representations of Q, K, and V, respectively, while the bottom row shows the resulting pulsed attention output. Each column represents a different condition: (a) in the figure represents the healthy control group in SESA; (b) represents the patient group in SESA; (c) represents the healthy control group in SSA; and (d) represents the patient group in SSA. In this pulse visualization, the white area represents the firing state of the binary pulse (value 1), while the black area represents the neuron being in a silent or non-firing state (value 0). Comparative analysis shows that, driven by the EEG signals of the healthy control group, the LIF spiking neurons in the corresponding time dimension of the SESA and SSA modules in the architecture of this invention exhibit significantly stronger pulse activation density. This intuitive display based on binary pulse sequences provides an event-driven representation with physical meaning, distinct from the continuous numerical characteristics of traditional artificial neural networks, thus deeply verifying the inherent interpretability advantage of this invention in processing complex neurophysiological signals.

[0081] The above interpretability analysis results show that the method proposed in this invention, while maintaining high recognition performance, can reveal its modeling method of spatiotemporal features of EEG signals in a visual form, and has good interpretability.

[0082] In summary, although the technical solutions of the present invention have been described in detail with reference to the preferred embodiments shown in the accompanying drawings, those skilled in the art will readily understand that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles and spirit of the present invention, those skilled in the art can make various substitutions, modifications, and variations to the related technical features. For example, specific components such as memory or neuron models in the architecture of the present invention can be replaced with other equivalent architectures or components. The embodiments of the present invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, equivalent substitutions, improvements, or changes made within the spirit and principles of the present invention should be considered as having been taught in conjunction with the description herein and should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing electroencephalogram (EEG) signals using a spatiotemporal joint spiking neural network, characterized in that, Specifically, the steps include the following: Step 1: Organize the continuous EEG signals of the subject into a multidimensional data representation that includes both channel spatial information and time sequence information, and pulse-encode the multidimensional data representation to generate a binary pulse sequence covering multiple pulse time steps; Step 2: Input the binary pulse sequence into a multi-scale spiking neural network feature extraction architecture for spatiotemporal feature extraction. The specific feature extraction mechanism includes: introducing a learnable latent encoding vector as a unified query and broadcasting it to each pulse time step; accumulating the pulse query signal in the time dimension to extract temporal evolution features, and accumulating the pulse value signal in the channel dimension to extract channel correlation features; combining the two based on element-wise pulse operation to generate EEG self-attention feature output; simultaneously performing joint operation on the temporal continuity of the pulse sequence and the spatial topological relationship of the multi-channel EEG signal to generate a spatiotemporally enhanced pulse representation; and obtaining multi-scale high-level spatiotemporal pulse features through iterative execution and downsampling processing of the above mechanism. Step 3: After feature extraction is completed, the obtained high-level spatiotemporal pulse features are input into the classification module. The classification module aggregates the features of each time step through global average pooling, obtains the category prediction results corresponding to each time step through fully connected mapping, and outputs the final recognition and classification results of the EEG signal by averaging the prediction results of all time steps.

2. The spatiotemporal joint spiking neural network EEG signal recognition method as described in claim 1, characterized in that, The pulse encoding process in step one is as follows: by utilizing the membrane potential accumulation and firing decay mechanism of spiking neurons in the spiking neural network, the input continuous EEG signal is converted into a binary sequence over multiple pulse time steps to preserve the relative amplitude relationship and temporal dynamic characteristics of the signal.

3. The spatiotemporal joint spiking neural network EEG signal recognition method as described in claim 1, characterized in that, In step one, before organizing the subject's EEG signal into a multidimensional data representation, the EEG signal is preprocessed. The preprocessing operation includes filtering out power frequency noise, low-frequency drift, and artifact interference.

4. The spatiotemporal joint spiking neural network EEG signal recognition method as described in claim 1, characterized in that, In step two, the multi-scale spiking neural network feature extraction architecture includes three stages of downsampling and feature extraction: the first stage sequentially performs spiking attention calculation based on latent encoding vectors and spatiotemporal spiking joint enhancement calculation; the second stage sequentially performs spiking EEG self-attention calculation, spatiotemporal spiking joint enhancement calculation, spiking attention calculation based on latent encoding vectors, and spatiotemporal spiking joint enhancement calculation again; the third stage continuously performs three sets of standard spiking self-attention calculation and spatiotemporal spiking joint enhancement calculation; when moving from the previous stage to the next stage, the input spiking features are downsampled in the channel dimension and time dimension.

5. The spatiotemporal joint spiking neural network EEG signal recognition method as described in claim 1, characterized in that, The method also includes a step for visualizing EEG features: statistically analyzing the pulse firing intensity, temporal distribution, and channel response differences induced by different types of EEG signals within the feature extraction architecture; Based on the difference data, an impulse response distribution map containing response characteristics in both spatial and temporal dimensions is generated, and the distribution map is output to a display device to provide an intuitive explanation of the model's decision-making.

6. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images