Classification method of spiking neural network based on multi-scale frequency perception
By using multi-scale spatiotemporal fusion and global information fusion modules, the spatiotemporal modeling capability of spiking neural networks is improved, solving the problem of insufficient accuracy in existing technologies and achieving efficient feature capture and accurate classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO UNIV
- Filing Date
- 2025-12-09
- Publication Date
- 2026-04-21
AI Technical Summary
Existing spiking neural networks lag behind deep neural networks in terms of accuracy, and existing methods fail to fully exploit their temporal dynamic characteristics, resulting in shortcomings in capturing fine-grained spatiotemporal dynamic information and global information.
A multi-scale spatiotemporal fusion module was designed, which integrates features through multi-scale spatial and temporal windows. Combined with a global information fusion module, Hartley transform and adaptive filtering kernel are used to achieve efficient information fusion and feature capture.
It significantly improves the classification accuracy and overall performance of spiking neural networks in complex dynamic scenes, while reducing computational complexity and the number of model parameters.
Smart Images
Figure BDA0005729649960000071 
Figure FDA0005729649950000021
Abstract
Description
Technical Field
[0001] This invention relates to a classification method based on spiking neural networks, and more particularly to a classification method based on multi-scale frequency sensing spiking neural networks. Background Technology
[0002] Deep neural networks have demonstrated superior performance across numerous fields; however, their high computational cost and energy consumption make their deployment in edge computing, mobile devices, and other scenarios particularly challenging in practical applications. In contrast, spiking neural networks (SNNs) are renowned for their low power consumption and efficient temporal processing capabilities. These characteristics give SNNs significant potential and advantages in edge computing, brain-computer interfaces, and robotics. As a third-generation neural network, while SNNs have shown great advantages in many areas, their accuracy remains significantly lower than that of deep neural networks due to limitations in their network structure and optimization strategies. To address this issue, recent research has attempted to improve SNN performance by borrowing advanced architectures from deep neural networks or reducing information loss and gradient errors. By introducing convolutional neural networks and residual learning strategies, these methods have enabled deeper network construction and improved the accuracy of SNNs. However, these methods utilize only single-scale information and fail to fully exploit the temporal dynamics of SNNs, resulting in significant limitations in capturing fine-grained spatiotemporal dynamics and global information. Some studies have enhanced the ability of spiking neural networks to handle complex patterns by introducing the global information fusion capability of the attention mechanism. However, due to the inherent quadratic complexity of the attention mechanism, the computational cost has increased significantly. Furthermore, some methods reduce gradient error by adaptively adjusting the shape of the activation function to approximate a step function during training. While these methods have their advantages, they may also lead to vanishing or exploding gradients in the later stages of training due to overly steep activation function shapes. Summary of the Invention
[0003] The technical problem to be solved by this invention is to provide a classification method based on multi-scale frequency-aware spiking neural networks that fully leverages the spatiotemporal modeling potential of spiking neural networks. This method can more accurately understand and analyze input content in complex dynamic scenes, thereby significantly improving the overall performance of the model.
[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a classification method based on a multi-scale frequency sensing spiking neural network, comprising the following steps:
[0005] Step 1: Select N samples from the original database to form a training set. Set up a pulse segmentation module. The pulse segmentation module segments the training samples in each training set into feature vectors containing multiple different spatial scales and arranges them in descending order of spatial scale to form a multi-scale feature set. The features output by the pulse segmentation module represent the multi-scale spatiotemporal characteristics of the original input image.
[0006] Step 2: Construct the spiking neural network classification model to be trained, including a multi-scale spatial fusion module, a multi-scale temporal fusion module, a global information fusion module, and a classification head. Select the last specified number of m feature vectors from the multi-scale feature set and input them into the multi-scale spatial fusion module, where 2 ≤ m ≤ 8. The multi-scale spatial fusion module includes a spatial convolutional layer, a first-order normalization layer, an upsampling layer, and a pooling layer. The spatial convolutional layer convolves the input feature vectors to obtain convolutional features, and the number of channels of the m convolutional features is mapped to the same set size. Then, the m convolutional features are fed into the first-order normalization layer for normalization to obtain the normalized features of each convolutional feature. The upsampling layer upsamples and fuses the m normalized features in sequence to obtain their respective fused feature maps. The pooling layer applies max pooling to adjust all fused feature maps to the same size to obtain the multi-scale spatial fusion features.
[0007] Step 3: The multi-scale temporal fusion module contains m parallel temporal processing branches for processing a single multi-scale spatial fusion feature. Each temporal processing branch has a first spiking neuron, and each first spiking neuron corresponds to a time window size. The specific relationship between the time window size and the spatial scale of the multi-scale spatial fusion feature processed by the temporal processing branch is as follows: the branch that processes multi-scale spatial fusion features with smaller spatial scales has a larger time window size. The division of the time window ensures that each spiking neuron layer can independently process inputs at different time scales, enabling the network to capture the dynamic patterns of the input data at different time scales. The multi-scale spatial fusion features are divided according to the time window size, and the corresponding first spiking neurons process them to obtain temporal features. Subsequently, the temporal features are subjected to convolution and layer normalization in the temporal dimension, which can linearly transform the features at different time steps and extract the correlations. Then, all the processed features are stacked in order and averaged along the stacking dimension to output the multi-scale spatiotemporal fusion feature. This nonlinear mapping process from low-level semantic information to high-level semantic information is regarded as another form of temporal information and integrated into the SNN to fully exploit the temporal modeling capabilities of the SNN.
[0008] Step 4: Input the multi-scale spatiotemporal fusion features into the global information fusion module. The global information fusion module includes a pulse layer and an adaptive filtering module. The specific processing procedure is as follows:
[0009] Step 4-1: The pulse layer transforms the multi-scale spatiotemporal fusion features into pulse features, converts the dot product into a masking operation to further improve model efficiency, and transforms the pulse features into frequency domain features through Hartley transform.
[0010] Step 4-2: Generate the corresponding adaptive frequency domain filter kernel based on the frequency domain characteristics through an adaptive branch consisting of a second spiking neuron, a second linear layer, and a second batch normalization layer;
[0011] Step 4-3: Perform Hadamard product operation on the frequency domain features and the adaptive frequency domain filter kernel to obtain the frequency domain filter features; achieve efficient global information fusion;
[0012] Step 4-4: Perform an inverse Hartley transform on the frequency domain filtered features to obtain spatial domain features that incorporate global information;
[0013] This adaptive frequency domain filter kernel adjusts adaptively according to the input to optimize the response to key frequency components, thereby enhancing the capture of important features;
[0014] Step 5: Input the frequency domain filtered features into the classification head, which consists of global average pooling and fully connected layers, and perform forward propagation to generate the corresponding classification results;
[0015] Step 6: Define the loss function of the spiking neural network classification model to be trained and set the maximum number of iterations. Use the backpropagation algorithm to train and optimize the spiking neural network classification model until the set maximum number of iterations is reached, then stop the training process to obtain the trained spiking neural network classification model.
[0016] Step 7: Select any image to be classified from the original database and input it into the trained spiking neural network classification model. The trained spiking neural network classification model outputs the score of the image to be classified in each category. The category with the highest score is taken as the category of the image to be classified, thus completing the classification process.
[0017] Compared with existing technologies, the advantages of this invention lie in its specially designed multi-scale spatiotemporal fusion module, which fully utilizes the spatiotemporal dynamic characteristics of spiking neural networks. This module employs a top-down architecture to integrate multi-scale spatial features and further integrates temporal features through multi-scale time windows, thereby fully leveraging the spatiotemporal modeling potential of spiking neural networks. The multi-scale spatiotemporal fusion module can capture high-frequency local details and low-frequency global features in the input. This fusion method enables spiking neural networks to more accurately understand and analyze input content in complex dynamic scenes, thus significantly improving the overall performance of the model.
[0018] This invention introduces the convolution theorem into spiking neural networks for the first time and designs a lightweight global information fusion module adapted to the characteristics of spiking neural networks. This module first uses the Hartley transform to convert the pulse signal to the frequency domain. Then, it adaptively generates a filter kernel for the input in the frequency domain and reduces the number of model parameters and computational complexity by approximating large-scale dynamic convolution operations in the spatial domain through the Hadamard product. The global information fusion module not only efficiently integrates pulse features but also introduces an innovative global information fusion strategy for spiking neural network-based models.
[0019] Preferably, in step 3, the firing threshold of the first spiking neuron is... i and membrane potential decay factor i Adjustments are made based on the time scale of the frequency domain features being processed, specifically satisfying the following relationship: threshold i =base th +β th log(1+i), decay factor i =base df +γ df log(1+i), where i represents the index of the i-th multi-scale spatial fusion feature among m multi-scale spatial fusion features, and base th It is the base value of the excitation threshold, β th It is a scaling factor corresponding to the excitation threshold, base df It is the base value of the membrane potential decay factor, γ df Is with base df The corresponding scaling factors. In shallow networks, we use smaller time windows, lower spiking neuron thresholds, and smaller decay factors, which allows the SNN to quickly respond to small or subtle changes in the input, thus capturing high-frequency information. As the model depth increases, the time window gradually expands, and the threshold and decay factor of the spiking neuron also increase accordingly, enabling the network to capture more stable or global features, such as overall shape and continuous object motion.
[0020] Preferably, in step 6, during the forward propagation of training, the output signal S′[t] of all spiking neurons is composed of the fusion of the pulse signal and the real-valued signal of the substitution function:
[0021] S′[t]=(1-α)·Hea(H[t]-vth)+α·SF(H[t]-vth), where Hea(·) is the step function, SF(·) is the substitution function, H[t] is the current membrane potential, vth is the excitation threshold, and α is the fusion weight coefficient; the fusion weight coefficient α starts from the initial value of 1 and gradually decays to 0 with training iterations.
[0022] Preferably, the decay of α follows the following rule: Among them, iterations now The current total number of iterations, iterations is the total number of iterations within a training epoch, and epoch is the current total number of iterations. end This refers to the training cycle number at which the smooth transition optimization strategy ends. The proposed smooth transition optimization strategy decomposes the inherent discrete optimization problem in spiking neural networks into a series of subtasks, thereby minimizing information loss and gradient error during training. This strategy fuses the real-valued signal output by the substitution function with the pulse signal, gradually increasing the proportion of the pulse signal until a complete transition to a standard spiking neuron is achieved. The smooth transition optimization strategy does not alter the shape of the substitution function, ensuring a relatively smooth gradient in the membrane potential even in the later stages of training. Detailed Implementation
[0023] The present invention will now be described in further detail.
[0024] A classification method based on a multi-scale frequency-sensing spiking neural network includes the following steps:
[0025] Step 1: Select N samples from the original database to form a training set. Set up a pulse segmentation module. The pulse segmentation module segments the training samples in each training set into feature vectors containing multiple different spatial scales and arranges them in descending order of spatial scale to form a multi-scale feature set. The features output by the pulse segmentation module represent the multi-scale spatiotemporal characteristics of the original input image.
[0026] Step 2: Construct the spiking neural network classification model to be trained, including a multi-scale spatial fusion module, a multi-scale temporal fusion module, a global information fusion module, and a classification head. Select the last specified number of m feature vectors from the multi-scale feature set and input them into the multi-scale spatial fusion module (2≤m≤8). The multi-scale spatial fusion module includes a spatial convolutional layer, a first-order normalization layer, an upsampling layer, and a pooling layer. The spatial convolutional layer convolves the input feature vectors to obtain convolutional features, and the number of channels of the m convolutional features is mapped to the same set size. Then, the m convolutional features are fed into the first-order normalization layer for normalization to obtain the normalized features of each convolutional feature. The upsampling layer upsamples and fuses the m normalized features in sequence to obtain their respective fused feature maps. The pooling layer applies max pooling to adjust all fused feature maps to the same size to obtain multi-scale spatial fusion features. By integrating information from different layers, the richness and completeness of features are enhanced, and the information loss that may be caused by a single scale is reduced.
[0027] Step 3: The multi-scale temporal fusion module contains m parallel temporal processing branches for processing a single multi-scale spatial fusion feature. Each temporal processing branch has a first spiking neuron, and each first spiking neuron has a corresponding time window size. The specific relationship between the time window size and the spatial scale of the multi-scale spatial fusion feature processed by the temporal processing branch is as follows: the branch that processes multi-scale spatial fusion features with smaller spatial scales has a larger time window size. The division of the time window ensures that each spiking neuron layer can independently process inputs at different time scales, enabling the network to capture the dynamic patterns of the input data at different time scales. The multi-scale spatial fusion features are divided according to the time window size, and the corresponding first spiking neuron processes them to obtain temporal features. Subsequently, the temporal features are subjected to convolution and layer normalization in the temporal dimension, which can perform linear transformations on features at different time steps and extract the correlations. Then, all processed features are stacked sequentially and averaged along the stacking dimension to output the multi-scale spatiotemporal fusion feature. This nonlinear mapping process from low-level semantic information to high-level semantic information is regarded as another form of temporal information and integrated into the network to fully exploit the network's temporal modeling capabilities.
[0028] Among them, the firing threshold of the first spiking neuron i and membrane potential decay factor i Adjustments are made based on the time scale of the frequency domain features being processed, specifically satisfying the following relationship: threshold i =base th +β th log(1+i), decay factor i =base df +γ df log(1+i), where i represents the index of the i-th multi-scale spatial fusion feature among m multi-scale spatial fusion features, and base th It is the base value of the excitation threshold, β th It is a scaling factor corresponding to the excitation threshold, base df It is the base value of the membrane potential decay factor, γ df Is with base df The corresponding scaling factor.
[0029] Step 4: Input the multi-scale spatiotemporal fusion features into the global information fusion module. The global information fusion module includes a pulse layer and an adaptive filtering module. The specific processing procedure is as follows:
[0030] Step 4-1: The pulse layer transforms the multi-scale spatiotemporal fusion features into pulse features, converts the dot product into a masking operation to further improve model efficiency, and transforms the pulse features into frequency domain features through Hartley transform.
[0031] Step 4-2: Generate the corresponding adaptive frequency domain filter kernel based on the frequency domain characteristics through an adaptive branch consisting of a second spiking neuron, a second linear layer, and a second batch normalization layer.
[0032] Step 4-3: Perform Hadamard product operation on the frequency domain features and the adaptive frequency domain filter kernel to obtain the frequency domain filter features; achieve efficient global information fusion.
[0033] Step 4-4: Perform an inverse Hartley transform on the frequency domain filtered features to obtain spatial domain features that incorporate global information.
[0034] This adaptive frequency domain filter kernel adjusts itself according to the input to optimize the response to key frequency components, thereby enhancing the capture of important features.
[0035] Step 5: Input the frequency domain filtered features into the classification head, which consists of global average pooling and fully connected layers, and propagate it forward to generate the corresponding classification results.
[0036] Step 6: Define the loss function of the spiking neural network classification model to be trained and set the maximum number of iterations. Use the backpropagation algorithm to train and optimize the spiking neural network classification model until the set maximum number of iterations is reached, then stop the training process to obtain the trained spiking neural network classification model.
[0037] During the forward propagation of training, the output signal S′[t] of all spiking neurons is composed of the fusion of the spiking signal and the real-valued signal of the substitution function:
[0038] S′[t]=(1-α)·Hea(H[t]-vth)+α·SF(H[t]-vth), where Hea(·) is the step function, SF(·) is the substitution function, H[t] is the current membrane potential, vth is the excitation threshold, and α is the fusion weight coefficient; the fusion weight coefficient α starts from an initial value of 1 and gradually decays to 0 with training iterations, and the decay of α follows the following rules: Among them, iteration now The current total number of iterations, iterations is the total number of iterations within a training epoch, and epoch is the current total number of iterations. end The training cycle number at which the optimization strategy for a smooth transition ends.
[0039] Step 7: Select any image to be classified from the original database and input it into the trained spiking neural network classification model. The trained spiking neural network classification model outputs the score of the image to be classified in each category. The category with the highest score is taken as the category of the image to be classified, thus completing the classification process.
[0040] To further evaluate the effectiveness of the present invention, Table 1 shows the performance comparison results of the classification method based on multi-scale frequency sensing spiking neural network (hereinafter referred to as the method) used in this example with existing methods on the Cifar10 / Cifar100 dataset.
[0041] Table 1
[0042]
[0043] On CIFAR10, our method achieves an accuracy of 96.33% with 8.01M parameters, surpassing the baseline model of Spikingformer-CML (95.95% accuracy, 9.33M parameters) and RealSpike (95.60% accuracy, 12.63M parameters). On the more complex CIFAR100 dataset, our method (81.5% accuracy, 8.01M parameters) shows a 1.1% improvement over the baseline model of Spikingformer-CML (80.4% accuracy, 9.37M parameters) and outperforms the ANN method Transformer-4-384 with the same structure (81.02% accuracy, 9.37M parameters).
[0044] This method achieves state-of-the-art top-1 accuracy on both datasets and has the smallest number of model parameters. Spiking Transformer-LD represents a model with L encoder blocks and D channels.
Claims
1. A classification method based on a multi-scale frequency-sensing spiking neural network, characterized in that... Includes the following steps: Step 1: Select N samples from the original database to form a training set, set up a pulse segmentation module, and use the pulse segmentation module to segment the training samples in each training set into feature vectors containing multiple different spatial scales and arrange them in order of spatial scale from large to small to form a multi-scale feature set. Step 2: Construct the spiking neural network classification model to be trained, including a multi-scale spatial fusion module, a multi-scale temporal fusion module, a global information fusion module, and a classification head. Select the last specified number of m feature vectors from the multi-scale feature set and input them into the multi-scale spatial fusion module, where 2 ≤ m ≤ 8. The multi-scale spatial fusion module includes a spatial convolutional layer, a first-order normalization layer, an upsampling layer, and a pooling layer. The spatial convolutional layer convolves the input feature vectors to obtain convolutional features, and the number of channels of the m convolutional features is mapped to the same set size. Then, the m convolutional features are fed into the first-order normalization layer for normalization to obtain the normalized features of each convolutional feature. The upsampling layer upsamples and fuses the m normalized features in sequence to obtain their respective fused feature maps. The pooling layer applies max pooling to adjust all fused feature maps to the same size to obtain the multi-scale spatial fusion features. Step 3: The multi-scale temporal fusion module contains m parallel temporal processing branches for processing a single multi-scale spatial fusion feature. Each temporal processing branch has a first spiking neuron, and each first spiking neuron has a corresponding time window size. The specific relationship between the time window size and the spatial scale of the multi-scale spatial fusion feature processed by the temporal processing branch is as follows: the branch that processes multi-scale spatial fusion features with smaller spatial scales has a larger time window size. The multi-scale spatial fusion feature is divided according to the time window size, and the corresponding first spiking neuron processes it to obtain the temporal feature. Subsequently, the temporal features are convolutional and layer normalized in the temporal dimension; then all the processed features are stacked in order and averaged along the stacking dimension to output multi-scale spatiotemporal fusion features. Step 4: Input the multi-scale spatiotemporal fusion features into the global information fusion module. The global information fusion module includes a pulse layer and an adaptive filtering module. The specific processing procedure is as follows: Step 4-1: The pulse layer transforms the multi-scale spatiotemporal fusion features into pulse features, and then uses Hartley transform to convert the pulse features to the frequency domain to obtain frequency domain features; Step 4-2: Generate the corresponding adaptive frequency domain filter kernel based on the frequency domain characteristics through an adaptive branch consisting of a second spiking neuron, a second linear layer, and a second batch normalization layer; Step 4-3: Perform a Hadamard product operation between the frequency domain features and the adaptive frequency domain filter kernel to obtain the frequency domain filter features; Step 4-4: Perform an inverse Hartley transform on the frequency domain filtered features to obtain spatial domain features that incorporate global information; Step 5: Input the frequency domain filtered features into the classification head, which consists of global average pooling and fully connected layers, and perform forward propagation to generate the corresponding classification results; Step 6: Define the loss function of the spiking neural network classification model to be trained and set the maximum number of iterations. Use the backpropagation algorithm to train and optimize the spiking neural network classification model until the set maximum number of iterations is reached, then stop the training process to obtain the trained spiking neural network classification model. Step 7: Select any image to be classified from the original database and input it into the trained spiking neural network classification model. The trained spiking neural network classification model outputs the score of the image to be classified in each category. The category with the highest score is taken as the category of the image to be classified, thus completing the classification process.
2. The classification method based on a multi-scale frequency-sensing spiking neural network according to claim 1, characterized in that... In step 3, the firing threshold of the first spiking neuron is... i and membrane potential decay factor i Adjustments are made based on the time scale of the frequency domain features being processed, specifically satisfying the following relationship: threshold i =base th +β th log(1+i), decay factor i =base df +γ df log(1+i), where i represents the index of the i-th multi-scale spatial fusion feature among m multi-scale spatial fusion features, and base th It is the base value of the excitation threshold, β th It is a scaling factor corresponding to the excitation threshold, base df It is the base value of the membrane potential decay factor, γ df Is with base df The corresponding scaling factor.
3. The classification method based on a multi-scale frequency-sensing spiking neural network according to claim 1, characterized in that... In step 6, during the forward propagation of training, the output signal S′[t] of all spiking neurons is composed of the fusion of the pulse signal and the real-valued signal of the substitution function: S′[t]=(1-α)·Hea(H[t]-vth)+α·SF(H[t]-vth), where Hea(·) is the step function, SF(·) is the substitution function, H[t] is the current membrane potential, vth is the excitation threshold, and α is the fusion weight coefficient; the fusion weight coefficient α starts from the initial value of 1 and gradually decays to 0 with training iterations.
4. The classification method based on a multi-scale frequency-sensing spiking neural network according to claim 3, characterized in that, The decay of α follows these rules: Among them, iterations now The current total number of iterations, iterations, and epochs represent the total number of iterations within a training epoch. end The training cycle number at which the optimization strategy for a smooth transition ends.