Neuromorphic data classification method based on gated space-time self-attention mechanism and spiking neural network

Abstract

The invention discloses a neuromorphic data classification method based on a gating space-time self-attention mechanism and a spiking neural network, and the method comprises the following steps: 1, selecting a public neuromorphic data set, carrying out the data preprocessing, and dividing the data into a training set, a verification set and a test set; 2, constructing a pulse neural network model to perform neuromorphic data classification, and introducing a gating space-time self-attention mechanism into the model to adaptively capture a global dependency relationship of space and time dimensions and enhance the space-time information representation capability of the model; and 3, training the constructed spiking neural network model by using the training set and the verification set, testing the trained model by using the samples in the preprocessed test set after training is completed, and outputting a prediction classification result of the samples by the model. According to the method, the representation capability of the spiking neural network in spatial-temporal feature modeling is remarkably enhanced through a gating spatial-temporal self-attention mechanism, and the precision of a neuromorphic data classification task is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision application technology, specifically relating to a neuromorphic data classification method based on a gated spatiotemporal self-attention mechanism and a spiking neural network. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep neural networks have achieved remarkable success in fields such as computer vision. However, traditional artificial neural networks typically require enormous computing resources and energy consumption, which limits their application in resource-constrained scenarios such as mobile devices and embedded systems. Spiking neural networks, due to their event-driven characteristics and biological rationale, are considered the third generation of neural networks. They transmit and process information at time steps through discrete pulse sequences, possessing great potential for low power consumption, and are particularly suitable for neuromorphic computing platforms.

[0003] Despite the high energy efficiency of spiking neural networks (SNNs), their performance in practical applications, especially when handling complex spatiotemporal information tasks, often lags behind mature artificial neural networks. To improve the performance of SNNs, particularly their spatiotemporal modeling capabilities, various methods have been proposed. One approach employs multi-scale temporal feature fusion, integrating multi-scale temporal information by combining feature vectors obtained from different convolution or pooling operations, or using decomposed convolutions to expand the receptive field to model global dependencies. However, these methods are limited by local receptive fields, making it difficult to model global temporal relationships between cues or motions, and the translation invariance of 3D convolutions can corrupt the phase sensitivity of time series.

[0004] Another category of methods attempts to mine global information using coupled spatiotemporal learning approaches. Some methods capture complex spatiotemporal dependencies by accumulating historical spatial information; others construct Transformer architectures to model rich global spatiotemporal dependencies on local features. However, these methods often implicitly capture long-range spatiotemporal relationships, leading the model to focus more on strongly correlated spatial features (such as edges and textures) while underestimating dynamic temporal motion information, thus failing to fundamentally solve the problem of spatiotemporal representation imbalance.

[0005] Therefore, there is an urgent need in this field for a mechanism that can explicitly and adaptively model long-range dependencies in the spatial and temporal dimensions, and effectively suppress irrelevant information and enhance feature selectivity, in order to solve the inherent spatiotemporal representation imbalance problem of spiking neural networks, thereby comprehensively improving their performance in demanding visual tasks. Summary of the Invention

[0006] To overcome the spatiotemporal imbalance in spatiotemporal feature representation in existing spiking neural networks (SNs) and the limitations of existing attention mechanisms in explicitly and efficiently modeling long-range spatiotemporal dependencies, this invention proposes a neuromorphic data classification method based on a gated spatiotemporal self-attention mechanism and spiking neural networks. This method aims to explicitly capture global spatial and temporal dependencies, adaptively fuse spatiotemporal features, and filter noise through a bio-inspired gating operation, thereby significantly improving the feature representation capability and performance of spiking neural networks in neuromorphic vision tasks.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A neuromorphic data classification method based on a gated spatiotemporal self-attention mechanism and a spiking neural network includes the following steps:

[0009] Step A: Select publicly available datasets containing different neuromorphic patterns, encode each sample in the dataset into a 0 / 1 impulse event and perform data preprocessing, and then divide it into training set, validation set and test set;

[0010] Step B: Construct a spiking neural network model and introduce a gated spatiotemporal self-attention mechanism into the spiking neural network model to enhance the network's spatiotemporal feature representation capability. This gated spatiotemporal self-attention mechanism includes a gated spatial self-attention module, a gated temporal self-attention module, and a feature fusion module.

[0011] Step C: Use the training and validation sets divided in Step A to train the spiking neural network model with gated spatiotemporal self-attention mechanism constructed in Step B, and optimize the model parameters to adapt to classification tasks of different neuromorphic datasets.

[0012] Step D: After preprocessing the samples in the test set, input them into the model trained in Step C for testing. The model outputs the predicted classification result of the sample and evaluates the accuracy.

[0013] In step B:

[0014] The gated spatial self-attention module is used to calculate the correlation between any two spatial elements in the input feature vector X∈R. T,B,C,H,W T represents the time step, B represents the input batch, C represents the input dimension, and H and W represent the height and width of the input sample, respectively. First, 1×1 pointwise convolution is applied for feature dimensionality reduction to generate a mapping vector, which is then transformed into a spatial query vector Q. S ∈R T×B,C / 8,H×W Q S and its transpose vector space key vector K S ∈R T×B,H×W,C / 8Perform a SoftMax transformation to form the spatial attention matrix S∈R T×B,H×W,H×W Each element S of the spatial attention matrix S ji The calculation formula is as follows:

[0015]

[0016] In the formula, S ji Q represents the degree of influence between positions i and j. Si Represents the spatial query vector Q S The i-th element, K Sj Represents the spatial key vector K S The j-th element, exp(Q Si ·K Sj ) represents element Q Si With K Sj The exponential calculation between them; using spiking neurons to gate the spatial attention matrix S, and then multiplying it by the original S, yields:

[0017] S G =S·LIF(S)

[0018] In the formula, LIF represents a spiking neuron. Its unique thresholding mechanism sets values ​​below the threshold in the spatial attention matrix S to 0 and values ​​above the threshold to 1, forming a binary gating mask. Multiplying this mask by S retains the values ​​above the threshold in the original spatial attention matrix S. G This represents a spatial attention matrix containing only highly similar features; the input feature vector X is convolved pointwise using a 1×1 method and then transformed to generate a spatial value vector V. S ∈R T×B,C,H×W And compare it with the spatial attention matrix S containing only high similarity features. G Weighted aggregation is performed, and finally, spatially enhanced features (SA) are generated through residual connections. Each element of SA... j The calculation formula is:

[0019]

[0020] In the formula, α is a parameter initialized to 0, used for adaptively learning feature importance, and S Gj S represents a spatial attention matrix containing only highly similar features. G The j-th element, V Si Represents the spatial value vector V S The i-th element, X jThis represents the j-th element of the input feature vector X; each position in the spatial augmentation feature SA is an adaptive weighted sum of global context information in the spatial dimension; residual connections preserve the original spatial details and can more flexibly combine features at different levels;

[0021] The gated temporal self-attention module is used to calculate the correlation between any two time steps in the input feature vector, for an input feature vector X∈R. T,B,C,H,W First, a dimensional transformation is performed to convert it into a time query vector Q. T ∈R C,B,T,H×W Q T and its transpose vector, time key vector K T ∈R C,B,H×W,T Perform a SoftMax transformation to form the time attention matrix T. M ∈R C,B,T,T The time attention matrix T M Each element T Mji The calculation formula is as follows:

[0022]

[0023] In the formula, T Mji This represents the degree of influence of time step i on time step j; a higher value indicates a greater weight of attention. Q Ti Represents the time query vector Q T The i-th element, K Tj Represents the time key vector K T The j-th element, exp(Q Ti ·K Tj ) represents element Q Ti With K Tj Exponential calculation between them; utilizing spiking neurons for the time attention matrix T M Perform gated filtering, then compare with the original T M Multiplying, we get:

[0024] T G =T M ·LIF(T M )

[0025] In the formula, LIF represents a spiking neuron, whose unique thresholding mechanism will affect the temporal attention matrix T. M Values ​​below the threshold are set to 0, and values ​​above the threshold are set to 1, forming a binary gating mask, which is then compared with T. M Multiplying them will preserve the original temporal attention matrix T. M Values ​​above the threshold, T G This represents the time attention matrix containing only similarity features; it transforms the input feature vector X to generate the time value vector V. T∈R T,B,C,H,W And compare it with the time attention matrix T containing only high similarity features. G Weighted aggregation is performed, and finally, the temporal augmentation feature TA is output through residual connections. Each element of TA... j The calculation formula is:

[0026]

[0027] In the formula, β is a parameter initialized to 0, used for adaptively learning feature importance, and T Gj T represents the time attention matrix containing only high-similarity features. G The j-th element, V Ti Represents the time value vector V S The i-th element, X j This represents the j-th element of the input feature vector X; it directly calculates the correlation between any two time steps, unaffected by the distortion caused by convolution, and preserves the original temporal correlation through residual connections;

[0028] The feature fusion module adaptively weights and fuses the output features from the spatial and temporal paths using a learnable scale parameter.

[0029] X output =γ·TA+(1-γ)·TA

[0030] In the formula, γ is a learnable scaling parameter, initially set to 0, X output This represents the attention features of the final output; the feature fusion module helps the spiking neural network automatically learn the optimal balance between spatial attention and temporal attention. Initially, γ=0 allows spatial attention to be gradually integrated, thereby enhancing the interaction and representation of spatiotemporal features.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] First, this invention explicitly and separately captures global spatial dependence and long-range temporal dependence through parallel spatial and temporal self-attention paths, fundamentally solving the problem of spatiotemporal representation imbalance in spiking neural networks, where spatial features dominate and temporal features decay.

[0033] Second, this invention innovatively utilizes the threshold gating mechanism unique to spiking neurons to set activation values ​​with low similarity in the attention matrix to zero, effectively filtering out irrelevant information and noise interference, and retaining only the most critical associated features.

[0034] Third, this invention adaptively fuses the output features of spatial and temporal paths through learnable scale parameters, enabling the network to automatically learn and optimize the optimal contribution ratio of spatial and temporal information according to the specific task and the characteristics of the input data, thus achieving more flexible and efficient multimodal feature fusion.

[0035] Fourth, while bringing significant performance improvements, this invention introduces only limited additional computational costs, fully considering the high efficiency requirements of spiking neural networks, achieving an excellent balance between performance and efficiency, and possessing the potential for practical deployment and application. Attached Figure Description

[0036] Figure 1 This is an overall flowchart of the method of the present invention.

[0037] Figure 2 This is a flowchart of the spiking neural network based on gated spatiotemporal self-attention in this invention.

[0038] Figure 3 This is a flowchart of the gated spatial self-attention module of the present invention.

[0039] Figure 4 This is the flowchart of the gated time self-attention method of the present invention.

[0040] Figure 5 This invention provides visualization heatmaps of key regions in different input images based on spiking neural network models with gated spatial self-attention, gated temporal self-attention, and gated spatiotemporal self-attention. Detailed Implementation

[0041] The specific details of each step of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] This invention proposes a spiking neural network classification method based on gated spatiotemporal self-attention. The entire process of this method is as follows: Figure 1 As shown.

[0043] This method mainly includes the following steps:

[0044] Step A: Select publicly available datasets containing different neuromorphic patterns, encode each sample in the dataset into a 0 / 1 impulse event and perform data preprocessing, and then divide it into training set, validation set and test set;

[0045] Step B: Construct a spiking neural network model, and as follows Figure 2 As shown, a gated spatiotemporal self-attention mechanism is introduced into the spiking neural network model to enhance the network's spatiotemporal feature representation capability. This gated spatiotemporal self-attention mechanism includes a gated spatial self-attention module, a gated temporal self-attention module, and a feature fusion module.

[0046] In step B:

[0047] The gated spatial self-attention module is used to calculate the correlation between any two spatial elements in the input features, such as... Figure 3 As shown, for the input feature vector X∈R T,B,C,H,W T represents the time step, B represents the input batch, C represents the input dimension, and H and W represent the height and width of the input sample, respectively. First, 1×1 pointwise convolution is applied for feature dimensionality reduction to generate a mapping vector, which is then transformed into a spatial query vector Q. S ∈R T×B,C / 8,H×W Q S and its transpose vector space key vector K S ∈R T ×B,H×W,C / 8 Perform a SoftMax transformation to form the spatial attention matrix S∈R T×B,H×W,H×W Each element S of the spatial attention matrix S ji The calculation formula is as follows:

[0048]

[0049] In the formula, S ji Q represents the degree of influence between positions i and j. Si Represents the spatial query vector Q S The i-th element, K Sj Represents the spatial key vector K S The j-th element, exp(Q Si ·K Sj ) represents element Q Si With K Sj The exponential calculation between them; using spiking neurons to gate the spatial attention matrix S, and then multiplying it by the original S, yields:

[0050] S G =S·LIF(S)

[0051] In the formula, LIF represents a spiking neuron. Its unique thresholding mechanism sets values ​​below the threshold in the spatial attention matrix S to 0 and values ​​above the threshold to 1, forming a binary gating mask. Multiplying this mask by S retains the values ​​above the threshold in the original spatial attention matrix S. G This represents a spatial attention matrix containing only highly similar features; the input feature vector X is convolved pointwise using a 1×1 method and then transformed to generate a spatial value vector V. S ∈R T×B,C,H×W And compare it with the spatial attention matrix S containing only high similarity features. G Weighted aggregation is performed, and finally, spatially enhanced features (SA) are generated through residual connections. Each element of SA... j The calculation formula is:

[0052]

[0053] In the formula, α is a parameter initialized to 0, used for adaptively learning feature importance, and S Gj S represents a spatial attention matrix containing only highly similar features. G The j-th element, V Si Represents the spatial value vector V S The i-th element, X j This represents the j-th element of the input feature vector X; each position in the spatial augmentation feature SA is an adaptive weighted sum of global context information in the spatial dimension; residual connections preserve the original spatial details and can more flexibly combine features at different levels;

[0054] The gated temporal self-attention module is used to calculate the correlation between any two time steps in the input feature vector, such as... Figure 4 As shown, for the input feature vector X∈R T,B,C,H,W First, a dimensional transformation is performed to convert it into a time query vector Q. T ∈R C ,B,T,H×W Q T and its transpose vector, time key vector K T ∈R C,B,H×W,T Perform a SoftMax transformation to form the time attention matrix T. M ∈R C,B,T,T The time attention matrix T M Each element T Mji The calculation formula is as follows:

[0055]

[0056] In the formula, T Mji This represents the degree of influence of time step i on time step j; a higher value indicates a greater weight of attention. Q Ti Represents the time query vector Q T The i-th element, K Tj Represents the time key vector K T The j-th element, exp(Q Ti ·K Tj ) represents element Q Ti With K Tj Exponential calculation between them; utilizing spiking neurons for the time attention matrix T M Perform gated filtering, then compare with the original T M Multiplying, we get:

[0057] T G =T M ·LIF(TM )

[0058] In the formula, LIF represents a spiking neuron, whose unique thresholding mechanism will affect the temporal attention matrix T. M Values ​​below the threshold are set to 0, and values ​​above the threshold are set to 1, forming a binary gating mask, which is then compared with T. M Multiplying them will preserve the original temporal attention matrix T. M Values ​​above the threshold, T G This represents the time attention matrix containing only similarity features; it transforms the input feature vector X to generate the time value vector V. T ∈R T,B,C,H,W And compare it with the time attention matrix T containing only high similarity features. G Weighted aggregation is performed, and finally, the temporal augmentation feature TA is output through residual connections. Each element of TA... j The calculation formula is:

[0059]

[0060] In the formula, β is a parameter initialized to 0, used for adaptively learning feature importance, and T Gj T represents the time attention matrix containing only high-similarity features. G The j-th element, V Ti Represents the time value vector V S The i-th element, X j This represents the j-th element of the input feature vector X; it directly calculates the correlation between any two time steps, unaffected by the distortion caused by convolution, and preserves the original temporal correlation through residual connections;

[0061] The feature fusion module adaptively weights and fuses the output features from the spatial and temporal paths using a learnable scale parameter.

[0062] X output =γ·SA+(1-γ)·TA

[0063] In the formula, γ is a learnable scaling parameter, initially set to 0, X output This represents the attention features of the final output; the feature fusion module helps the spiking neural network automatically learn the optimal balance between spatial attention and temporal attention. Initially, γ=0 allows spatial attention to be gradually integrated, thereby enhancing the interaction and representation of spatiotemporal features.

[0064] Step C: Use the training and validation sets divided in Step A to train the spiking neural network model with gated spatiotemporal self-attention mechanism constructed in Step B, and optimize the model parameters to adapt to classification tasks of different neuromorphic datasets.

[0065] Step D: After preprocessing the samples in the test set, input them into the model trained in Step C for testing. The model outputs the predicted classification result of the sample and evaluates the accuracy.

[0066] Experimental verification and result analysis

[0067] Experimental Objective: The embodiments of this invention aim to verify the effectiveness of the gated spatiotemporal self-attention mechanism in improving the performance of spiking neural networks. Through comparative experiments and ablation studies, it can significantly improve model performance on various neuromorphic datasets, exhibiting stronger feature representation capabilities and higher recognition accuracy.

[0068] Experimental Setup: To comprehensively evaluate the performance of this invention, experiments were conducted on several publicly available neuromorphic datasets, including CIFAR10-DVS, DVS-Gesture, and N-Caltech101 datasets. This invention employs a widely used VGG11-based spiking neural network as the backbone network. All experiments were conducted using the PyTorch framework, trained on eight NVIDIA Quadro RTX8000 GPUs with input resolutions of 128×128, 128×128, and 180×240, a batch size of 32, and the Adam optimizer. The initial learning rate was 0.001 with cosine decay, and the iterations were 1024.

[0069] Experimental Results and Analysis:

[0070] This invention uses the classification accuracy on the test set as a metric to measure model performance, and its calculation formula is as follows:

[0071]

[0072] The classification accuracy of the spiking neural network on the CIFAR10-DVS, DVS-Gesture and N-Caltech101 datasets is 81.3%, 98.2% and 80.88%, respectively. After adding the gated spatiotemporal self-attention mechanism, the classification accuracy is 83.7%, 99.31% and 82.81%, respectively, indicating that the gated spatiotemporal self-attention mechanism can effectively enhance the long-term modeling and spatiotemporal representation capabilities of the spiking neural network.

[0073] This invention uses ablation experiments to verify the effectiveness of different modules. By adding a gated spatial self-attention module, a gated temporal self-attention module, and a feature fusion module to the spiking neural network in sequence, the accuracy was 82.0%, 82.4%, and 83.5%, respectively, which improved the accuracy by 0.71%, 1.1%, and 2.2%. This shows that these three sub-modules can improve the classification performance of the spiking neural network.

[0074] This invention employs gradient-weighted class activation mapping (GFRP) to visualize key regions in an input image, such as... Figure 5 As shown, the features mainly appear at object boundaries and in high-frequency texture regions, indicating the effectiveness of the model in feature extraction. Compared with spiking neural networks, the model with gated spatial self-attention, gated temporal self-attention, and feature fusion modules pays more attention to the target category than the background, further verifying the ability of each module to enhance the discrimination of spatiotemporal features and suppress irrelevant background information.

Claims

1. A neuromorphic data classification method based on a gated spatio-temporal self-attention mechanism and a spiking neural network, characterized in that: The method comprises the following steps: Step A: selecting a public dataset containing different neuromorphic data, encoding each sample in the dataset into 0 / 1 pulse events and pre-processing the data, and then dividing the data into a training set, a validation set and a test set; Step B: constructing a pulse neural network model and introducing a gated spatio-temporal self-attention mechanism into the pulse neural network model to enhance the network's spatio-temporal feature representation capability, the gated spatio-temporal self-attention mechanism comprising a gated spatial self-attention module, a gated temporal self-attention module and a feature fusion module; Step C: training the pulse neural network model containing the gated spatio-temporal self-attention mechanism constructed in step B using the training set and the validation set divided in step A, and optimizing the model parameters to adapt to the classification task of different neuromorphic data sets; Step D: pre-processing the samples in the test set and inputting them into the model trained in step C for testing, the model outputting the predicted classification result of the samples and evaluating the accuracy.

2. The neuromorphic data classification method based on the gated spatio-temporal self-attention mechanism and the spiking neural network according to claim 1, characterized in that: In step B: Gated spatial self-attention module is used to calculate the correlation between any two spatial elements in the input feature, for input feature vector X ∈ R T,B,C,H,W , T represents time step, B represents input batch, C represents input dimension, H and W represent height and width of input sample respectively; first apply 1*1 point convolution for feature dimension reduction, generate mapping vector and then perform dimension transformation to spatial query vector Q S ∈ R T×B,C / 8,H×W ; Q S and its transpose vector spatial key vector K S ∈ R T×B,H×W,C / 8 are subjected to SoftMax transformation to form spatial attention matrix S ∈ R T×B,H×W,H×W , wherein each element S ji of spatial attention matrix S is calculated according to the following formula: In the formula, S ji represents the influence degree of position i and position j, Q Si represents the i-th element of the spatial query vector Q S , K Sj represents the j-th element of the spatial key vector K S , and exp(Q Si ·K Sj ) represents the exponential calculation between elements Q Si and K Sj ; the spatial attention matrix S is filtered by using the pulse neuron, and then multiplied by the original S to obtain the calculation result: S G = S · LIF(S) In the formula, LIF represents a pulse neuron, and its unique threshold mechanism will set the values below the threshold in the spatial attention matrix S to 0 and the values above the threshold to 1, forming a binary gated mask, and then multiplied by S, the values above the threshold in the original spatial attention matrix S will be retained, S G represents a spatial attention matrix containing only high-similarity features; the input feature vector X is subjected to 1 1 point-by-point convolution and feature transformation to generate a spatial value vector V S ∈R T×B,C,H×W , and is weighted and aggregated with the spatial attention matrix S G containing only high-similarity features, and finally a spatial enhancement feature SA is generated through a residual connection, each element SA j of which is calculated according to the formula: In the formula, α is a parameter initialized to 0, used for adaptively learning feature importance, and S Gj S represents a spatial attention matrix containing only highly similar features. G The j-th element, V Si Represents the spatial value vector V S The i-th element, X j This represents the j-th element of the input feature vector X; each position in the spatial augmentation feature SA is an adaptive weighted sum of global context information in the spatial dimension; residual connections preserve the original spatial details and can more flexibly combine features at different levels; The gating time self-attention module is used to calculate the correlation between any two time steps in the input feature vector, for an input feature vector X ∈ R T,B,C,H,W , a dimension transformation is first performed to obtain a time query vector Q T ∈ R C,B,T,H×W ; the Q T and its transpose vector time key vector K T ∈ R C,B,H×W,T are subjected to SoftMax transformation to form a time attention matrix T M ∈ R C,B,T,T , wherein each element T M of the time attention matrix T Mji is calculated according to the following formula: In the formula, T Mji represents the influence degree of time step i on time step j, the higher the value, the greater the attention weight, Q Ti represents the i-th element of the time query vector Q T , K Tj represents the j-th element of the time key vector K T , exp(Q Ti ·K Tj ) represents the exponential calculation between elements Q Ti and K Tj ; the time attention matrix T M is filtered by using the pulse neuron, and then multiplied by the original T M to obtain: T G = T M · LIF(T M ) In the formula, LIF represents a pulse neuron, and its unique threshold mechanism will set the time attention matrix T M to 0 below the threshold value and to 1 above the threshold value, forming a binary gated mask, which is multiplied with T M , so that the original time attention matrix T M with values above the threshold value is retained. G represents a time attention matrix containing only similarity features; the input feature vector X is subjected to feature transformation to generate a time value vector V T ∈R T,B,C,H,W , and is weighted and aggregated with the time attention matrix T G containing only high similarity features, and finally the time enhancement feature TA is output through a residual connection, each element TA j of which is calculated according to the formula: where β is a parameter initialized to 0 for adaptively learning the importance of features, T Gj represents the time attention matrix T G that only contains high similarity features Ti represents the i-th element of the time value vector V S represents the j-th element of the input feature vector X j represents the j-th element of the input feature vector X; directly calculates the correlation between any two time steps, which is not affected by the distortion caused by convolution, and retains the original time correlation through the residual connection; The feature fusion module adaptively weights and fuses the output features of the spatial path and the temporal path through a learnable scale parameter: X output = γ · SA + (1 - γ) · TA where γ is a learnable scale parameter, initialized to 0, X output denotes the attention feature for the final output. The feature fusion module helps the pulse neural network to automatically learn the optimal balance between spatial attention and temporal attention, and initially γ=0 enables the spatial attention to gradually integrate, thereby enhancing the interaction and representation of spatio-temporal features.

Images