Synaptic Convolutional Spiking Neural Network for Event Stream Classification
By introducing a multi-threshold LIF model with learnable attenuation coefficients in the pulsed neural network, combined with the output layer of the DVS data compression and voting mechanism, the time dimension and redundantness of event stream classification in the prior art are solved. The problem of the rest of the data enables lower latency and higher precision event stream classification.
Patent Information
- Application Number
- CN202210172647.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-24
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2042-02-24
AI Technical Summary
Existing pulsed neural networks are difficult to take into account the processing time dimension and a large amount of redundant data when processing event stream classification, resulting in an increase in delay, and it is almost impossible to complete effective classification through deep pulsed neural networks.
A synaptic convolutional pulse neural network for event flow classification is proposed, using a multi-threshold LIF model with learnable attenuation coefficients and a synaptic convolution layer based on synaptic model, combining DVS data compression input layer and voting mechanism The output layer, through data compression and precise synaptic model processing, shortens latency and improves accuracy.
On the premise of ensuring processing accuracy, the delay of the pulse neural network is effectively shortened, the length of the input pulse sequence is reduced, and the accuracy of the pulse neural network is improved.
Smart Images
Figure CN114548290B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a neural network, in particular to a synaptic convolutional pulse neural network for event stream classification. Background Art
[0002] The spiking neural network is a brain-inspired neural network model. As the third generation of neural networks, this model is inspired by the brain's neuron mechanism and has event-driven characteristics. It has great potential in extracting spatiotemporal features and is therefore very suitable for frame-based event stream classification tasks.
[0003] Dynamic vision sensors (DVS) represent visual information by using sparse and asynchronous events, and are often used to generate spike data for event stream classification tasks of spiking neural networks. However, due to the high temporal resolution and irregular dynamic changes of scenes, DVS sensors usually have a large amount of redundant data in the time dimension when recording events, which causes delays in spiking neural networks while ensuring accuracy, making event stream classification almost impossible to complete with deep spiking neural networks. Summary of the invention
[0004] The purpose of the present invention is to solve the technical problem that the pulse neural network currently used for event stream classification is difficult to take into account the processing time dimension and a large amount of redundant data, and to propose a synaptic convolutional pulse neural network for event stream classification, which can effectively shorten the delay of the pulse neural network while ensuring the processing accuracy.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] A synaptic convolutional spike neural network for event stream classification, which is special in that: it is built based on a multi-threshold LIF model with a learnable decay coefficient, and the spike neural network includes a DVS data compression input layer, multiple intermediate hidden layers and an output layer arranged in sequence;
[0007] The DVS data compression input layer compresses the pulse sequence with the specified compression rate by using the weighted pulse superposition method, and converts the input information into a real value of 0-1 by using the normalization method, and transmits it to the next layer of network as the input current;
[0008] The intermediate hidden layer includes a synaptic convolution layer based on a synaptic model and a fully connected layer based on a multi-threshold pulse neuron in layer A; A and B are both positive integers;
[0009] The synaptic convolution layer is composed of a first-order synaptic model and a convolution function, and includes a convolution layer, a batch normalization layer, a synaptic layer, and an average pooling layer arranged in sequence;
[0010] The synaptic model in the spike neuron model of the intermediate hidden layer adopts a zero-order model;
[0011] The output layer is based on a voting mechanism, which is used to randomly assign labels to neurons in the previous layer. The outputs of neurons with the same label are accumulated and averaged as the output pulse of the label.
[0012] Furthermore, the first-order synaptic model is:
[0013] I syn [t] = tanh(ω syn )I syn [t-1]+In[t]
[0014] in:
[0015] I syn [t] and I syn [t-1] is the output current of the synapse at time t and t-1;
[0016] ω syn is the current decay weight of the synapse;
[0017] In[t] is the input current of the synapse at time t.
[0018] Furthermore, the multi-threshold LIF model is:
[0019]
[0020] u i [t] = sigmoid(ω decay ) i [t-1]+I[t]+u reset [t]
[0021]
[0022] in:
[0023] I[t] is the synaptic input current sent by the previous layer of spiking neural network to the i-th spiking neuron;
[0024] i represents the i-th spiking neuron of the spiking neural network;
[0025] j represents the jth spiking neuron in the previous spiking neural network;
[0026] N is the number of spiking neurons in the previous layer of spiking neural network;
[0027] ω ij is the weight between the jth spiking neuron in the previous spiking neural network and the ith spiking neuron in the current spiking neural network;
[0028] s j [t] represents the output pulse of the jth spiking neuron in the previous spiking neural network at time t;
[0029] u i [t] is the membrane voltage of the ith spiking neuron at time t;
[0030] u i [t-1] is the membrane voltage of the ith spiking neuron at time t-1;
[0031] u reset [t] is the reset voltage of the spiking neuron at time t;
[0032] ω decay is the decay weight of the spiking neuron;
[0033] s i [t] is the output pulse of the i-th spiking neuron at time t;
[0034] S max is the upper limit of the output pulse of the spiking neuron;
[0035] V th is the threshold voltage of the spiking neuron.
[0036] Furthermore, in the DVS data compression input layer: data is described in the form of pulse events, a pulse sequence with a specified compression rate is subjected to data accumulation compression, and the coordinate information, occurrence time and polarity of the pulse time of the pulse event are obtained.
[0037] Furthermore, the coordinate information, occurrence time and polarity of the pulse time of the pulse event are obtained by dividing the pulse data into T parts according to the time interval of the pulse event in the DVS data. q The time interval of the data is [q a ,q b ], in each piece of data, the data accumulation and compression are performed according to the following formula;
[0038]
[0039]
[0040]
[0041] Among them, floor() is the rounding function, q a and q b is the lower and upper bounds of the time interval of the qth data; T event is the time when the pulse event occurs; N ris the resolution factor; k=0,1,...,N r ;
[0042] E(x m ,y m ,t m ,p m ) represents the coordinate information, occurrence time and polarity of the pulse event m;
[0043] F(q,p,x,y) represents the value of the data with polarity p at the (x,y) position in the qth group of data after compression.
[0044] Furthermore, the synaptic convolution layer uses normal distribution to initialize the synaptic current decay weight ω according to formula (8): syn Build:
[0045] ω syn ~[0,0.05].
[0046] Furthermore, the construction of the fully connected layer based on multi-threshold pulse neurons is specifically as follows:
[0047] Initialize all the decay weights of the spiking neurons to 0;
[0048] Then, according to formula (9), the normal distribution is used to initialize the inter-layer weights of the pulse neural network;
[0049] W (l) ~[V th / N l-1 , / 0.5]
[0050] Among them, W (l) is the weight of the l-th layer of the pulse neural network, and the elements in the matrix are denoted by ω ij , V th is the voltage threshold of the spiking neuron, N l-1 is the number of neurons in the previous layer of spiking neural network.
[0051] Furthermore, in the fully connected layer, all neurons are randomly divided into N class The average output of each group of neurons is used as an input of the output layer based on the voting mechanism; the classification result of the event flow corresponding to the maximum input value is selected in the output layer based on the voting mechanism.
[0052] Furthermore, the number of layers A of the synaptic convolution layer, the number of layers B of the fully connected layer, and the number of synapses or neurons in each layer can be increased or decreased according to the difficulty of the classification task, and it can be considered that the neural network has achieved the best recognition accuracy.
[0053] Compared with the prior art, the present invention has the following beneficial technical effects:
[0054] 1) The synaptic convolutional spike neural network for event stream classification proposed in the present invention replaces the traditional single-threshold LIF model with a multi-threshold LIF model with a learnable attenuation coefficient, and proposes a synaptic convolution layer based on the synaptic model, which effectively shortens the delay of the spike neural network while ensuring processing accuracy.
[0055] 2) The DVS data compression input layer in the synaptic convolutional spike neural network for event stream classification proposed in the present invention can effectively reduce the length of the input pulse sequence by compressing the time and space dimensions.
[0056] 3) The synaptic convolutional spike neural network for event stream classification proposed in the present invention effectively improves the accuracy of the spike neural network while shortening the delay of the spike neural network by introducing additional attenuation weights of the multi-threshold LIF model and the current attenuation weights in the synaptic model. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 A schematic diagram of a neural network architecture of an embodiment of a synaptic convolutional spike neural network for event stream classification according to the present invention;
[0058] Figure 2 It is a schematic diagram of a DVS data compression input layer in an embodiment of a synaptic convolutional spike neural network for event stream classification of the present invention;
[0059] Figure 3 This is a schematic diagram of a synaptic convolution layer based on synaptic convolution in a synaptic convolutional spike neural network for event stream classification according to the present invention. DETAILED DESCRIPTION
[0060] In order to make the objects, advantages and features of the present invention more clear, a synaptic convolutional spike neural network for event stream classification proposed by the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0061] like Figure 1 As shown, a spiking neural network based on synaptic convolution for event stream classification is shown. The traditional single-threshold LIF model is replaced by a multi-threshold LIF model, which is constructed based on a multi-threshold LIF model with a learnable attenuation coefficient. The spiking neural network includes an input layer, multiple intermediate hidden layers and an output layer arranged in sequence.
[0062] The input layer of the neural network is the DVS data compression input layer, which is used to transform the neural network information. A weighted pulse superposition model is used to convert the DVS data into an input current with a preset accuracy. Specifically, a weighted pulse superposition method is used to compress the pulse sequence with a specified compression rate, and a normalization method is used to convert the input information into a real value of 0-1, which is passed to the next layer of the network as the input current.
[0063] The middle hidden layer (i.e., multiple hidden layers after excluding the first input layer and the last output layer) includes a synaptic convolution layer based on a synaptic model in layer A and a fully connected layer based on multi-threshold pulse neurons in layer B.
[0064] The synaptic convolution layer is composed of a first-order synaptic model and a convolution function, and includes a convolution layer, a batch normalization layer, a synaptic layer, and an average pooling layer arranged in sequence;
[0065] The synaptic model in the spike neuron model of the intermediate hidden layer adopts a zero-order model;
[0066] The output layer is based on a voting mechanism, which is used to randomly assign labels to neurons in the previous layer. The outputs of neurons with the same label are accumulated and averaged as the output pulse of the label.
[0067] The specific steps to construct a spiking neural network are as follows:
[0068] Input Layer
[0069] like Figure 2 As shown in the DVS data compression input layer, DVS data is described in the form of pulse events, E(x m ,y m ,t m ,p m ) represents the coordinate information, occurrence time and polarity of the pulse event m.
[0070] First, the pulse data is divided into T parts according to the time interval of the pulse event in the DVS data. q The time interval of the data is [q a ,q b ], in each piece of data, data accumulation and compression are performed according to formula (3), and finally the compressed data is obtained.
[0071] DVS data compression calculation is shown in the following formula:
[0072]
[0073]
[0074]
[0075] Among them, floor() is the rounding function, q a and q b is the lower and upper bounds of the time interval of the qth data; T event is the time when the pulse event occurs; N r is the resolution factor; k=0,1,...,N r; F(q,p,x,y) represents the value of the data with polarity p at the (x,y) position in the qth group of data after compression.
[0076] Synaptic convolution layer based on synaptic model
[0077] like Figure 3 As shown, the synaptic convolution layer based on the synaptic model is composed of a first-order synaptic model and a convolution function, including a convolution layer, a batch normalization layer, a synaptic layer, and an average pooling layer arranged in sequence.
[0078] The first-order synaptic model is:
[0079] I syn [t] = tanh(ω syn )I syn [t-1]+In[t] (4)
[0080] in:
[0081] I syn [t] and I syn [t-1] is the output current of the synapse at time t and t-1;
[0082] ω syn is the current decay weight of the synapse;
[0083] In[t] is the input current of the synapse at time t.
[0084] In the synaptic convolution layer, the synaptic current decay weight ω is initialized using normal distribution according to formula (5): syn :
[0085] ω syn ~[0,0.05] (5)
[0086] Fully connected layer
[0087] The fully connected layer based on multi-threshold spiking neurons adopts a multi-threshold LIF model with learnable decay coefficients.
[0088] The multi-threshold LIF model is:
[0089]
[0090] u i [t] = sigmoid(ω decay ) i [t-1]+I[t]+u reset [t] (7)
[0091]
[0092] in:
[0093] I[t] is the synaptic input current sent by the previous layer of spiking neural network to the i-th spiking neuron;
[0094] i represents the i-th spiking neuron of the spiking neural network;
[0095] j represents the jth spiking neuron in the previous spiking neural network;
[0096] N is the number of spiking neurons in the previous layer of spiking neural network;
[0097] ω ij is the weight between the jth spiking neuron in the previous spiking neural network and the ith spiking neuron in the current spiking neural network;
[0098] s j [t] represents the output pulse of the jth spiking neuron in the previous spiking neural network at time t;
[0099] u i [t] is the membrane voltage of the ith spiking neuron at time t;
[0100] u i [t-1] is the membrane voltage of the ith spiking neuron at time t-1;
[0101] u reset [t] is the reset voltage of the spiking neuron at time t;
[0102] ω decay is the decay weight of the spiking neuron;
[0103] s i [t] is the output pulse of the i-th spiking neuron at time t;
[0104] S max is the upper limit of the output pulse of the spiking neuron;
[0105] V th is the threshold voltage of the spiking neuron.
[0106] In the fully connected layer based on multi-threshold spiking neurons, the decay weights of the spiking neurons are all initialized to 0; the inter-layer weights of the spiking neural network are initialized using normal distribution according to formula (9);
[0107] W (l) ~[V th / N l-1 ,0.5] (9)
[0108] Among them, W (l) is the weight of the l-th layer of the pulse neural network, and the elements in the matrix are denoted by ω ij , Vth is the voltage threshold of the spiking neuron, N l-1 is the number of neurons in the previous layer of spiking neural network.
[0109] In order to simplify the calculation, the synapse model in the pulse neuron model adopts a zero-order model, that is, I[t] in the model is calculated using a zero-order model.
[0110] Output Layer
[0111] In the output layer based on the voting mechanism, all neurons in the previous fully connected layer are randomly divided into N class Group, N class For the categories that need to be divided into classification problems, the average value of the output of each group of neurons is used as an input of the output layer based on the voting mechanism. The output layer based on the voting mechanism selects the category corresponding to the largest input value as the final classification result through comparison.
[0112] When the number of synaptic convolutional layers A, the number of fully connected layers B, and the number of synapses or neurons in each layer can be increased or decreased according to the difficulty of the classification task, it can be considered that the neural network has achieved the best recognition accuracy.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.
Claims
1. A synaptic convolutional spike neural network for event stream classification, characterized by: Based on the multi-threshold LIF model with learnable decay coefficients, the spiking neural network includes a DVS data compression input layer, multiple intermediate hidden layers, and an output layer arranged in sequence; The DVS data compression input layer compresses the pulse sequence with the specified compression rate by using the weighted pulse superposition method, and converts the input information into a real value of 0-1 by using the normalization method, and transmits it to the next layer of network as the input current; The intermediate hidden layer includes a synaptic convolution layer based on a synaptic model and a fully connected layer based on a multi-threshold pulse neuron in layer A; A and B are both positive integers; The synaptic convolution layer is composed of a first-order synaptic model and a convolution function, and includes a convolution layer, a batch normalization layer, a synaptic layer, and an average pooling layer arranged in sequence; The synaptic model in the spike neuron model of the intermediate hidden layer adopts a zero-order model; The output layer is based on a voting mechanism, which is used to randomly assign labels to neurons in the previous layer. The outputs of neurons with the same label are accumulated and averaged as the output pulse of the label. The first-order synaptic model is: I syn [t]=tanh(ω syn )I syn [t-1]+In[t] in: I syn [t] and I syn [t-1] is the output current of the synapse at time t and t-1; ω syn is the current decay weight of the synapse; In[t] is the input current of the synapse at time t; The multi-threshold LIF model is: u i [t]=sigmoid(ω decay )u i [t-1]+I[t]+u reset [t] in: I[t] is the synaptic input current sent by the previous layer of spiking neural network to the i-th spiking neuron; i represents the i-th spiking neuron of the spiking neural network; j represents the jth spiking neuron in the previous spiking neural network; N is the number of spiking neurons in the previous layer of spiking neural network; ω ij is the weight between the jth spiking neuron in the previous spiking neural network and the ith spiking neuron in the current spiking neural network; s j [t] represents the output pulse of the jth spiking neuron in the previous spiking neural network at time t; u i [t] is the membrane voltage of the ith spiking neuron at time t; u i [t-1] is the membrane voltage of the ith spiking neuron at time t-1; u reset [t] is the reset voltage of the spiking neuron at time t; ω decay is the decay weight of the spiking neuron; s i [t] is the output pulse of the i-th spiking neuron at time t; S max is the upper limit of the output pulse of the spiking neuron; V th is the threshold voltage of the spiking neuron.
2. The synaptic convolutional spike neural network for event stream classification according to claim 1, characterized in that: In the DVS data compression input layer: data is described in the form of pulse events, a pulse sequence with a specified compression rate is compressed for data accumulation, and the coordinate information, occurrence time and polarity of the pulse time of the pulse event are obtained.
3. The synaptic convolutional spike neural network for event stream classification according to claim 2, characterized in that: The coordinate information, occurrence time and polarity of the pulse time of the pulse event are obtained by dividing the pulse data into T parts according to the time interval of the pulse event in the DVS data. q The time interval of the data is [q a ,q b ], in each piece of data, the data accumulation and compression are performed according to the following formula; Among them, floor() is the rounding function, q a and q b is the lower and upper bounds of the time interval of the qth data; T event is the time when the pulse event occurs; N r is the resolution factor; k=0,1,...,N r ; E(x m ,y m ,t m ,p m ) represents the coordinate information, occurrence time and polarity of the pulse event m; F(q,p,x,y) represents the value of the data with polarity p at the (x,y) position in the qth group of data after compression.
4. The synaptic convolutional spike neural network for event stream classification according to claim 1, characterized in that: The synaptic convolution layer uses the normal distribution to initialize the synaptic current decay weight ω according to the following formula syn Build: oh syn ~[0,0.05].
5. The synaptic convolutional spike neural network for event stream classification according to claim 1, characterized in that: The construction of the fully connected layer based on multi-threshold pulse neurons is specifically as follows: Initialize all the decay weights of the spiking neurons to 0; Then, the inter-layer weights of the pulse neural network are initialized using normal distribution according to the following formula; W (l) ~[V th / N l-1 , / 0.5] Among them, W (l) is the weight of the l-th layer of the pulse neural network, and the elements in the matrix are denoted by ω ij , V th is the voltage threshold of the spiking neuron, N l-1 is the number of neurons in the previous layer of spiking neural network.
6. The synaptic convolutional spike neural network for event stream classification according to claim 5, characterized in that: In the fully connected layer, all neurons are randomly divided into N class The average output of each group of neurons is used as an input of the output layer based on the voting mechanism; the classification result of the event flow corresponding to the maximum input value is selected in the output layer based on the voting mechanism.
7. The synaptic convolutional spike neural network for event stream classification according to any one of claims 1 to 6, characterized in that: The number of layers A of the synaptic convolution layer, the number of layers B of the fully connected layer, and the number of synapses or neurons in each layer can be increased or decreased according to the difficulty of the classification task.