Event stream classification method based on temporal channel joint attention based on self-attention

Abstract

The present invention discloses an event stream classification method based on self-attention and time-channel joint attention, which belongs to the field of computer vision recognition technology. The implementation method of the present invention is as follows: pre-processing the neuromorphic data set, converting the data represented by the event stream into a frame-based representation; the time-channel joint attention module based on self-attention adopts a compression and excitation structure; in the compression stage, average pooling and maximum pooling are respectively used to aggregate spatial information and superimpose and merge them to obtain time-channel features; in the excitation stage, self-attention is used to globally model the time-channel features to obtain the global dependency of channel information in the time dimension; by training the pulse neural network of the time-channel joint attention of self-attention, the global context information in the time and channel dimensions is captured; the trained pulse neural network is used to realize event stream classification, thereby improving the accuracy of event stream classification. The present invention can improve the accuracy and efficiency of visual recognition.

Description

Technical Field

[0001] The present invention relates to an event stream classification method based on time channel joint attention of self-attention, and belongs to the technical field of computer vision recognition. Background Art

[0002] Spiking Neural Networks (SNNs) operate in an event-driven manner and are highly biomimetic, known as the third generation of artificial neural networks. Spiking neurons transmit information via discrete pulses, enabling them to process dynamic data in spatiotemporal patterns. This makes SNNs promising for applications requiring both real-time performance and energy efficiency.

[0003] Neuromorphic vision sensors, inspired by biological visual processing mechanisms, capture changes in light intensity within the field of view and generate asynchronous event streams. A representative example is the dynamic vision sensor (DVS), which encodes the timing, location, and polarity of each pixel's brightness changes into an event stream at an extremely high event rate. Compared to traditional frame-based cameras, event-based DVSs offer a very high dynamic range, enabling them to capture information under complex lighting conditions, while also providing high temporal resolution and avoiding motion blur. Because events are triggered only when intensity changes, DVSs require fewer resources. These advantages have fostered the application of DVSs in various vision tasks, such as autonomous driving, high-speed object tracking, optical flow estimation, and action recognition. Neuromorphic datasets captured by DVSs are event-based and inherently temporal, providing a promising platform for spiking neural networks to demonstrate their spatiotemporal information processing capabilities. However, they suffer from low classification accuracy in event streams.

[0004] Attention mechanisms simulate the human ability to selectively locate salient regions in complex scenes. Researchers have introduced attention mechanisms into SNNs as lightweight auxiliary units for easy integration with existing architectures, enhancing the expressive power of the basic model and thus improving classification accuracy. For example, the temporal attention model TA-SNN assigns an attention factor to each input frame to determine its importance to the final decision. The multidimensional attention model MA-SNN calculates attention weights across the three dimensions of time, channel, and space, using them to optimize membrane potential and regulate spike responses. STSC-SNN combines temporal convolution with attention mechanisms to implement spatiotemporal synaptic connections, achieving synaptic filtering and gating functions, enhancing the spatiotemporal receptive field of synaptic connections and establishing temporal dependencies across layers. SCTFA establishes a spatial-channel-temporal fusion attention module that effectively captures potential target regions by leveraging historically accumulated spatial channel information. TCJA-SNN uses one-dimensional convolution to model attention in both the temporal and channel dimensions. However, these convolutions limit the spatiotemporal receptive field and fail to establish global temporal and channel dependencies. Summary of the Invention

[0005] In order to solve the problem of low accuracy of existing event stream classification, the main purpose of the present invention is to provide an event stream classification method based on self-attention and time-channel joint attention, use the self-attention mechanism to perform global modeling in the time and channel dimensions, and implement event stream classification based on pulse neural network with time-channel joint attention to improve the accuracy of event stream classification.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] The event stream classification method based on time channel joint attention and self-attention disclosed in the present invention includes the following steps:

[0008] Step 1: Preprocess the neuromorphic dataset and convert the event stream representation into a frame-based representation.

[0009] Step 2: The Self-attention-based Temporal Channel joint Attention (STCA) module adopts a compression and excitation structure to construct a spiking neural network based on self-attention. In the compression stage, average pooling and maximum pooling are used to aggregate spatial information and superimpose and merge them to obtain time-channel features. In the excitation stage, self-attention is used to globally model the time-channel features to obtain the global dependency of channel information in the time dimension.

[0010] The input of the temporal channel joint attention STCA module is Where T is the time step, C is the number of channels, H and W are rows and columns respectively; in the compression stage, the average pooling operation and the maximum pooling operation are used to aggregate spatial information to generate two different time-channel features, namely the average pooling feature and the maximum pooling feature, which are superimposed and merged to obtain:

[0011] R=AvgPool(X)+MaxPool(X), (1)

[0012] Among them, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, Represents time-channel characteristics;

[0013] In the excitation phase, the self-attention mechanism is used to obtain the global dependency of the channel information of the time-channel feature R in the time dimension. The query matrix Q, key matrix K, and value matrix V in the self-attention are calculated as follows:

[0014] Q=RW Q , K=RW K , V=RW V (2)

[0015] in, and They are the weight matrices of the query matrix Q, key matrix K, and value matrix V, C is the number of channels, and the output of self-attention The calculation is as follows:

[0016] A=softmax(QK T )V. (3)

[0017] The output of the self-attention-based temporal channel joint attention STCA module is expressed as:

[0018] X STCA =f⊙X, (4)

[0019] in, is the weight vector of STCA, σ is the sigmoid function, ⊙ represents element-wise multiplication, Represents the features extracted by the STCA module; the STCA module is plug-and-play and can be inserted after the convolution operation in the convolution-based spiking neural network and before the neurons aggregate spatiotemporal information to form a spiking neural network with time channel joint attention based on self-attention.

[0020] Step 3: Take the dataset preprocessed in step 1 as input and use the spatio-temporal backpropagation (STBP) algorithm to train and test the spiking neural network constructed in step 2; capture the global context information in time and channel dimensions by training the spiking neural network of self-attention and time-channel joint attention; realize event stream classification based on self-attention and time-channel joint attention through the trained spiking neural network to improve the accuracy of event stream classification.

[0021] Beneficial effects:

[0022] 1. The event stream classification method based on self-attention and time-channel joint attention disclosed in the present invention can globally model the time and channel dimension features by adding a time-channel joint attention module based on self-attention to the pulse neural network, capture the global context information of the time and channel dimensions, and construct a pulse neural network based on self-attention and time-channel joint attention; adopt the STBP algorithm to train and test the pulse neural network; realize event stream classification based on self-attention and time-channel joint attention through the trained pulse neural network, and improve the accuracy of event stream classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The temporal channel joint attention module based on self-attention disclosed in this invention;

[0024] Figure 2 Flowchart of the event stream classification method based on self-attention and time channel joint attention of the present invention.

[0025] Figure 3 Comparison of classification results of the event stream classification method based on self-attention and time channel joint attention on the CIFAR10-DVS dataset. DETAILED DESCRIPTION

[0026] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. The technical problems solved by the technical solution of the present invention and the beneficial effects thereof are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not serve to limit the present invention in any way.

[0027] The embodiment of the present invention provides an event stream classification method based on time channel joint attention of self-attention, which is used to improve classification accuracy, including steps S1 to S4:

[0028] S1. Obtain the neuromorphic dataset CIFAR10-DVS.

[0029] S2. Preprocess the CIFAR10-DVS dataset and convert the data represented by the event stream into a frame-based representation;

[0030] Integrating the original event stream form into frame form data is a common processing method. Specifically, the original event stream data is denoted as E(x i , y i , t i , p i ), 0 ≤ i ≤ N - 1. The event stream is evenly divided into T segments. Denote a certain frame in the integrated frame form data as F(j), and the pixel value at the (p, x, y) position is F(j, p, x, y), 0 ≤ j < T. F(j) is integrated from the events with indices between j l and j r in the event stream:

[0031]

[0032]

[0033]

[0034] Among them, is floor function, I p,x,y (p i , x i , y i ) is an indicator function, which takes the value of 1 if and only if (p, x, y) = (p i , x i , y i ), and 0 otherwise.

[0035] S3. Construct a spiking neural network based on self-attention-based temporal channel joint attention;

[0036] As Figure 1 shown, the self-attention-based temporal channel joint attention (STCA) module adopts a squeeze-and-excitation structure; in the squeeze stage, average pooling and max pooling are respectively used to aggregate spatial information and stack and merge them to obtain time-channel features; in the excitation stage, self-attention is used to globally model the time-channel features to obtain the global dependence of channel information in the time dimension; the specific steps are as follows:

[0037] Denote the input of the STCA module as Where T is the time step, C is the number of channels, H and W are rows and columns respectively; in the compression stage, the average pooling operation and the maximum pooling operation are used to aggregate spatial information to generate two different time-channel features, namely the average pooling feature and the maximum pooling feature, which are superimposed and merged to obtain:

[0038] R=AvgPool(X)+MaxPool(X), (4)

[0039] Among them, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, Represents time-channel characteristics;

[0040] In the excitation phase, the self-attention mechanism is used to obtain the global dependency of the channel information of the time-channel feature R in the time dimension. The query matrix Q, key matrix K, and value matrix V in the self-attention are calculated as follows:

[0041] Q=RW Q , K=RW K , V=RW V . (5)

[0042] in, and They are the weight matrices of the query matrix Q, key matrix K, and value matrix V, C is the number of channels, and the output of self-attention The calculation is as follows:

[0043] A=softmax(QK T )V. (6)

[0044] The weight vector of the STCA module can be obtained by using residual connection and sigmoid function for the self-attention module

[0045] f=σ(R+A) (7)

[0046] Where σ is the sigmoid function. Finally, the output of the STCA module is expressed as:

[0047] X STCA =f⊙X, (8)

[0048] Among them, ⊙ represents element-wise multiplication, Represents the features extracted by the STCA module.

[0049] The pulse network structure constructed in this embodiment is Input-Conv-BN-Neuron-MP-Conv-BN–STCA-Neuron-MP-Conv-BN-Neuron-MP-Conv-BN-STCA-Neuron-MP-DP-FC-Neuron-DP-FC-Neuron-Voting, where Conv represents the convolution layer, BN represents the batch normalization layer, DP represents dropout, Neuron adopts the PLIF neuron model, MP represents maximum pooling, and Voting represents the voting layer.

[0050] S4: Take the data set preprocessed in step S2 as input, and use the STBP algorithm to train and test the spiking neural network constructed in step S3 to obtain the optimal event stream classification result.

[0051] The event stream classification method based on time channel joint attention based on self-attention disclosed in this embodiment is trained in the environment of Intel(R) Core(TM) i7-8700CPU3.20GHz and NVIDIA RTX2080Ti GPU using SpikingJelly and Pytorch deep learning framework. The experimental parameters of network training are set as Batchsize 8, time step T 10, epochs 1000, learning rate 0.001, Adam optimizer, momentum 0.9, dropout 0.5, and pulse neuron parameter V th The STBP back propagation algorithm is used for training.

[0052] The event stream classification method based on time channel joint attention and self-attention disclosed in this embodiment has a classification accuracy of 74.40% under the above experimental conditions. Under the same experimental conditions, the classification result of the comparison algorithm TA-SNN is 73.70%, and the classification result of SNN without attention is 72.00%. The comparison results of training accuracy are shown in Figure 2. Figure 3 As shown, the dotted line represents the TA-SNN algorithm, the dotted line represents the algorithm without attention, and the solid line represents the event stream classification method disclosed in this embodiment. The analysis results show that the method of this embodiment can converge to a higher level of accuracy, verifying that the present invention can improve the classification accuracy of event streams.

[0053] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An event stream classification method based on time channel joint attention of self-attention, characterized by: The steps include: Step 1: Preprocess the neuromorphic dataset and convert the event stream representation into a frame-based representation. Step 2: The temporal-channel joint attention module based on self-attention adopts a compression and excitation structure to construct a spiking neural network based on temporal-channel joint attention based on self-attention. In the compression stage, average pooling and maximum pooling are used to aggregate spatial information and superimpose and merge them to obtain time-channel features. In the excitation stage, self-attention is used to globally model the time-channel features to obtain the global dependency of channel information in the time dimension. The implementation method of step 2 is: The input of the temporal channel joint attention STCA module is Where T is the time step, C is the number of channels, H and W are rows and columns respectively; in the compression stage, the average pooling operation and the maximum pooling operation are used to aggregate spatial information to generate two different time-channel features, namely the average pooling feature and the maximum pooling feature, which are superimposed and merged to obtain: R=AvgPool(X)+MaxPool(X), (1) Among them, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, Represents time-channel characteristics; In the excitation phase, the self-attention mechanism is used to obtain the global dependency of the channel information of the time-channel feature R in the time dimension. The query matrix Q, key matrix K, and value matrix V in the self-attention are calculated as follows: Q=RW Q , K=RW K , V=RW V . (2) in, and They are the weight matrices of the query matrix Q, key matrix K, and value matrix V, C is the number of channels, and the output of self-attention The calculation is as follows: A=softmax(QK T )V. (3) The output of the self-attention temporal channel joint attention STCA module is expressed as: X STCA =f⊙X, (4) in, is the weight vector of STCA, σ is the sigmoid function, ⊙ represents element-wise multiplication, Represents the features extracted by the STCA module; the STCA module is plug-and-play and can be inserted after the convolution operation in the convolution-based spiking neural network and before the neurons aggregate spatiotemporal information to form a spiking neural network based on self-attention and time-channel joint attention; Step 3: Take the dataset preprocessed in step 1 as input and use the spatio-temporal backpropagation (STBP) algorithm to train and test the spiking neural network constructed in step 2; capture the global context information in time and channel dimensions by training the spiking neural network based on time-channel joint attention of self-attention; realize event stream classification based on time-channel joint attention of self-attention through the trained spiking neural network to improve the accuracy of event stream classification.

Images