Picture recognition system based on pulse neural network with self-attention mechanism

Abstract

The picture recognition system based on the pulse neural network with a self-attention mechanism comprises an encoding module, an attention module and a decoding module.The encoding module is composed of a convolutional encoding layer and position encoding, receives an input picture sequence and generates a pulse sequence containing feature information of the input picture sequence.The attention module is composed of multiple layers of self-attention blocks, receives the pulse sequence, is processed through time and space self-attention operations respectively, and outputs a pulse sequence with richer time and space features.The decoding module receives the pulse sequence, compresses and extracts a final feature vector through sequence pooling, and inputs the feature vector into a classification head to obtain a picture recognition result.The application constructs a novel architecture based on the pulse neural network for picture recognition, and achieves the most advanced effect in the pulse neural network in the picture recognition task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and spiking neural network, and relates to an image recognition technology using artificial intelligence, in particular a spiking neural network, and in particular to a picture recognition system based on a spiking neural network with a self-attention mechanism. BACKGROUND

[0002] In the current network architecture for training spiking neural networks, traditional convolutional neural network architecture is mainly used. Due to the local receptive field characteristic of the convolutional neural network, a large receptive field can be finally obtained only by stacking many layers. The self-attention model proposed in recent years solves this problem. Due to the global receptive field brought by the self-attention operation, the self-attention model has a great impact on the field of natural language processing and has become a dominant method. Because of its strong ability to model long-distance connections, it has also been applied to the field of vision and has achieved quite effective results. However, due to the lack of inductive bias possessed by the convolutional neural network, a large amount of data is needed for training, and the suboptimality shown during training makes it difficult to train. The above factors have led to the fact that the self-attention model has not been applied to the field of spiking neural networks.

[0003] In the field of picture recognition, the models currently used are mostly convolutional neural networks and self-attention models based on traditional neural networks. However, these models face a trade-off between power consumption and accuracy in practical applications. High-precision models often have unacceptable power consumption, while models with relatively low power consumption often do not have high enough accuracy. Moreover, even the relatively low-power models have power consumption that is several orders of magnitude higher than the human brain. How to achieve picture recognition accuracy close to or even surpassing the human brain using the power consumption of the human brain is still a difficult technical problem to solve. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a picture recognition system based on a spiking neural network with a self-attention mechanism. The first purpose is to achieve high-precision picture recognition, and the second purpose is to reduce the power consumption of the recognition process or the hardware resources it relies on.

[0005] In order to achieve the above purpose, the technical solution adopted by the present application is:

[0006] The picture recognition system based on a spiking neural network with a self-attention mechanism comprises:

[0007] The encoding module converts the input picture sequence into a spike sequence through a convolutional encoding layer, and then adds learnable position encoding to obtain a picture spike sequence with spatial and temporal information;

[0008] The attention module is stacked by a plurality of layers of multi-head self-attention blocks, each layer of multi-head self-attention block is composed of layer normalization, time multi-head attention mechanism, space multi-head attention mechanism and multilayer perceptron; the layer normalization normalizes the picture pulse sequence with space-time information, stabilizes the learning process of the network; the time multi-head attention mechanism extracts features in the time dimension, the space attention mechanism extracts features in the space dimension, and the multilayer perceptron adds a nonlinear factor to the network and enhances the representation ability of the network;

[0009] The decoding module receives the pulse sequence processed by the attention module, obtains the final feature vector through sequence pooling, and finally sends the feature vector into a fully connected head for picture recognition.

[0010] The application further provides a picture recognition method using the picture recognition system based on the pulse neural network with a self-attention mechanism, and the method comprises the following steps:

[0011] Pictures are captured by using a camera to obtain any required number of pictures;

[0012] The pictures are normalized, and each picture is copied P times to form a picture sequence;

[0013] The processed pictures are sent into the encoding module in a sequence form, and the encoding and attention mechanism are performed to output a recognition result from the decoding module;

[0014] The recognition result is displayed.

[0015] The application further provides a picture recognition device, which comprises:

[0016] A picture acquisition part is used for acquiring pictures to be recognized;

[0017] A picture preprocessing part is used for normalizing the acquired pictures and copying the pictures P times to form a picture sequence;

[0018] A processing and storage part is used for storing and running the picture recognition system, taking the picture sequence obtained by the picture preprocessing part as input, and obtaining a recognition result;

[0019] A display part is used for displaying the recognition result and the original picture simultaneously.

[0020] Compared with the prior art, the application has the following beneficial effects:

[0021] 1. The method uses a pulse neural network based on a self-attention mechanism to provide a low-power and high-precision picture recognition method.

[0022] 2.The network structure of the method has a global receptive field, stronger spatiotemporal modeling capability, can more effectively use the inherent spatiotemporal characteristics of the pulse neuron, can better use limited parameters to capture information in the picture, and realizes a high-precision picture recognition system.

[0023] 3.Due to the leakage characteristics of the pulse neuron determined by the neural dynamics equation itself, the method shows a very low excitation frequency during actual picture recognition, and the network has no activity when not excited, the sparse pulse excitation characteristics of the entire network make only a very small part of it work, thereby realizing low-power picture recognition. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a schematic diagram for the overall overview of the present application.

[0025] Figure 2 is a schematic diagram of the convolutional encoding layer.

[0026] Figure 3 is a schematic diagram of the residual block in the original residual network.

[0027] Figure 4 is a schematic diagram of the attention block.

[0028] Figure 5 is a schematic diagram of the sequence pooling. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0030] The present application provides a picture recognition system based on a pulse neural network with a self-attention mechanism, adopts a new architecture for training the pulse neural network, promotes the development of the pulse neural network, and simultaneously uses the low-power characteristics of the pulse neural network to provide a high-precision, low-power model for picture recognition.

[0031] As shown in Figure 1 The picture recognition system of the present application adopts a structure of processing input by a convolutional encoding layer and extracting spatiotemporal information by a self-attention layer, which specifically includes:

[0032] An encoding module receives picture data and encodes it to form a pulse sequence. Specifically, the input picture sequence is converted into a pulse sequence by a convolutional encoding layer, and then a learnable position encoding is added to obtain a picture pulse sequence with spatiotemporal information.

[0033] The attention module is stacked by multiple layers of multi-head self-attention blocks, receives the pulse sequence, and is processed by time and space self-attention operations respectively, and outputs a pulse sequence with richer temporal and spatial features. Specifically, each multi-head self-attention block is composed of four modules of layer normalization, time multi-head attention mechanism, space multi-head attention mechanism and multilayer perceptron. The layer normalization normalizes the input vector to stabilize the learning process of the network, the time multi-head attention mechanism extracts the features in the time dimension, the space attention mechanism extracts the features in the space dimension, and the multilayer perceptron adds a nonlinear factor to the network to enhance the representation ability of the network.

[0034] The decoding module receives the pulse sequence, compresses and extracts the final feature vector through sequence pooling, and finally sends it to the output processor required by the corresponding task, such as the full connection head required by picture processing, to obtain the picture recognition result.

[0035] In the picture recognition system of the present application, the space-time attention mechanism capable of capturing rich space-time information is integrated in the system, which can more effectively utilize the inherent space-time characteristics of the pulse neuron, and capture the information in the picture by using limited parameters, thereby realizing high-precision picture recognition. At the same time, due to the leakage characteristics of the pulse neuron, the power consumption of the entire system is much lower than that of the conventional deep model.

[0036] In the embodiment of the present application, the encoding module includes a convolutional encoding layer and a position encoding. As shown in Figure 2 The convolutional encoding layer is a convolutional neural network designed for pulse neural networks, which can convert the input picture sequence into a pulse sequence, and the formulaic expression is as follows:

[0037] Conv(x)=SN(BN(Conv2d(x))),

[0038] CT d (x)=Conv(Conv(x))+Conv(AvgPool(x)),

[0039] CT(x)=Conv(Conv(x))+x

[0040] Where the input picture sequence is denoted as T represents the time dimension of the sequence, C represents the channel number of the picture, if it is an RGB picture, the channel number is 3, H and W represent the height and width of the picture respectively. Conv2d represents a conventional convolution operation, BN is a batch normalization layer, SN represents a pulse neuron, AvgPool represents an average pooling layer, CT d represents a convolutional encoding layer with down-sampling, and CT represents a conventional convolutional encoding layer without down-sampling.

[0041] Position coding is used to add the relative order information of space-time in the pulse sequence, so that the network structure has higher ability to process space-time information, which is formulated as follows:

[0042] z (p,t) = a (p,t) + e (p,t)

[0043] wherein D is the number of channels of the input picture sequence after the convolutional coding layer processing, and the vector after the convolutional coding layer processing is D, H' and W' are the number of channels, height and width of the processed picture features respectively. wherein N = H' x W', and a (p,t) represents the vector representation of a in the tth dimension in T and the pth dimension in N, represents the learnable code initialized to 0 at the corresponding space-time position.

[0044] In the embodiments of the present application, the membrane potential is used to represent the state of the pulse neuron in the convolutional coding layer and the self-attention block, and the state is updated using an ordinary differential equation, and the equation parameters are intracellular parameters with nonlinear time-varying characteristics, the state of the pulse neuron is updated by the current state and the influence of the input synapse, and the real-time membrane potential of the pulse neuron itself has a tendency to restore to the resting potential over time.

[0045] Specifically, the mathematical formula of the state equation corresponding to the pulse neuron is as follows:

[0046]

[0047] wherein H[t] and V[t] respectively correspond to the membrane potential of the pulse neuron after accepting the input at time step t = 1, 2,..., T and after triggering the pulse, and when t = 0, the neuron state is initialized to 0;

[0048] X[t] is the input from the previous layer network accepted at time step t, which is converted into a pulse sequence after being processed by the pulse neuron, and τ is the membrane time constant, which is defined as follows:

[0049]

[0050] C m represents the membrane capacitance of the pulse neuron, and g l is the membrane conductance of the pulse neuron;

[0051] The formula of the trigger pulse is as follows:

[0052] S[t] = Θ(H[t] - V th )

[0053] S[t] is the output pulse at time step t, and Θ(x) is a step function, i.e., Θ(x) = 1 when x > 0, and Θ(x) = 0 otherwise. th is the threshold of the firing pulse. For different kinds of pulse neurons, S[t] transmitted to the next layer is also different, for example, for the pulse-based and analog-based leaky integrate-and-fire discharge models, the input transmitted to the next layer is S[t] and ReLU(H[t]) respectively, where ReLU is a linear rectifier function that returns the original input when the input is greater than 0, and returns 0 when the input is less than or equal to 0.

[0054] The formula for updating H[t] after the trigger pulse to obtain V[t] is:

[0055] V[t] = H[t](1-S[t])+V reset S[t]

[0056] V reset is the reset potential after the trigger pulse.

[0057] wherein the firing function of the pulse neuron, i.e., the step function, cannot be directly back-propagated because of its non-differentiable nature during back-propagation, so an analog gradient function is used to replace its position in the computation graph, and the formula of the analog function is:

[0058]

[0059] After derivation, the derivative is:

[0060]

[0061] α is a hyperparameter artificially specified to control the slope of the analog gradient function. By replacing the step function with this function during back-propagation, the gradient can be successfully back-propagated, and the network can normally use the deep learning framework for back-propagation.

[0062] As Figure 4 shown, in the embodiments of the present application, both the time attention mechanism and the space attention mechanism are self-attention operations, the difference being that one operates in the time dimension and the other operates in the space dimension.

[0063] wherein the operation formula of the space self-attention mechanism is as follows:

[0064]

[0065] The operation formula of the time self-attention mechanism is as follows:

[0066]

[0067] l = 1,..., L represents the layer number to which the multi-head self-attention block belongs, h = 1,..., H represents the head to which the multi-head self-attention belongs, p / p' = 1,..., N represents the position of the vector in the spatial dimension, t / t' = 1,..., T represents the position of the vector in the time dimension, L, H, N, and T respectively represent the total number of layers of the multi-head self-attention block, the total number of heads of the multi-head self-attention block, the total spatial and time dimensions of the multi-head self-attention block. is the vector of the vector at the spatial position p and the time position t of the hth head of the lth multi-head self-attention block. Given the output z after the position encoding processing, k / q / v can be obtained by performing a linear transformation on z, SM(x) is a normalized exponential function for normalizing a given vector x into a probability distribution with a sum of 1, D h = D / H is the dimension of each head, where D is the dimension of the input vector.

[0068] Finally, the self-attention operation concatenates each head and performs a linear mapping to obtain the final result:

[0069]

[0070] s is the output vector with richer spatio-temporal representation after the time or spatial multi-head self-attention processing, Concat(x,...,y) is a concatenation function, W O is a linear mapping matrix.

[0071] The multi-layer perceptron is composed of a fully connected layer and a spiking neuron, which is formulated as follows:

[0072] z (l) = FC(SN(FC(s (l) ))

[0073] s (l) is the result of the input of the lth multi-head self-attention block after the time and spatial multi-head self-attention mechanism processing, z (l) is the final output of the lth multi-head self-attention block, FC represents a fully connected layer, and SN represents a spiking neuron.

[0074] In an embodiment of the present application, the decoding module includes a sequence pooling layer and a fully connected classification head.

[0075] As Figure 5 shown, the sequence pooling layer can actively learn to assign different weights to different sequence segment features, which is formulated as follows:

[0076] o = SM(FC(z (L) )) T z (L)

[0077] wherein L represents z (L) is the output of the last multi-head self-attention block, and finally the obtained o is sent into a fully connected classification head to obtain the final network picture recognition result.

[0078] In one specific embodiment of the present application, the application of the picture recognition system of the present application in the ImageNet picture recognition data set is described in detail.

[0079] ImageNet is a large picture recognition data set, which has a total of one thousand classes, and is divided into a training set and a validation set. The training set contains one million three hundred thousand pictures, and the validation set contains fifty thousand pictures. Each picture has a corresponding label.

[0080] Referring again to Figure 1 , the embodiment adopts an eight-layer convolutional neural network and a structure stacked with seven self-attention blocks, including:

[0081] The encoding module corresponds to the convolutional encoding layer and the position encoding, and accepts a picture format file. In order to adapt to the characteristics of the spiking neuron processing time series tasks, the picture is repeated multiple times to simulate a time series input into the convolutional encoding layer to convert the picture input into a pulse sequence. The multi-layer convolutional neural network can make the receptive field of the encoding layer larger, and the down-sampling step of the convolutional head can also be used to control the computational complexity, achieving a balance between accuracy and efficiency. Although the convolutional neural network and the spiking neuron contain spatial and temporal position information, adding position encoding does not cause negative effects, so the position encoding in the original self-attention model is retained. Referring to Figure 2 and Figure 3 , the convolutional neural network used is different from the convolutional neural network in the traditional residual network. When the network weights are initialized with all zeros, the original residual network can maintain the identity mapping for the activation function of the non-spiking neuron. However, due to the leakage characteristics of the spiking neuron, the input to the output will continuously decay, so the condition for achieving identity mapping is relatively harsh. After changing the connection mode, the input to the side branch connection will not be processed by the spiking neuron, so it will not decay, thereby achieving identity mapping and allowing the network to learn better. The convolutional neural network has four blocks, each with a convolution kernel size of (3, 3). The dimensions of each block are 64, 128, 256, and 512, respectively. Each block will down-sample the input once, reducing the encoding length of the final input to the self-attention block by four times each time.

[0082] The attention module, referring to Figure 4, including layer normalization, temporal attention mechanism, spatial attention mechanism and multi-layer perceptron. Layer normalization normalizes the input data, making the network training more stable, while reducing the impact of small training batch size. The temporal and spatial attention here are decomposed temporal and spatial attention. The temporal attention calculates the self-attention score of the target block and the block at the same position in different time steps, and the spatial attention calculates the self-attention score of the block at different positions in the same time step. The multi-layer perceptron includes two fully connected networks and a pulse neuron, which is used to add nonlinear elements to the network to make the network have stronger representation ability. The parameters of each layer are the same, 8 heads of attention are used, the magnification of the middle layer of the multi-layer perceptron is 3, and the dimension of each layer of encoding is 512.

[0083] The decoding module, with reference to Figure 5 , uses sequence pooling to compress the information processed by the attention module and sends it to the fully connected classification head. Since ImageNet has 1000 classes, the fully connected classification head has 1000 neurons. Finally, the obtained 1000-dimensional output is processed by the normalization exponential function to obtain the probability value to get the final classification result. In this case, each dimension of the output 1000-dimensional vector corresponds to the probability of the input feature vector being identified as the picture of that dimension. For example, (0.95, 0, 0.01, ···, 0.04), the largest component 0.95 in the vector corresponds to the picture of the first dimension in ImageNet, which means that the input feature vector is most likely to be the first picture in ImageNet for the system, so the corresponding recognition result is output, for example, the output result is: carp.

[0084] In the use of the picture recognition system of the present application, that is, the picture recognition method, the following steps are mainly included:

[0085] Step 1, use a camera or other device to capture and collect pictures. The number of pictures can be any value according to needs.

[0086] Step 2, pre-process the collected pictures, mainly including normalization processing and copying each picture P times to form a picture sequence. In an embodiment of the present application, the normalization processing can be: dividing the picture with pixel values from 0 to 255 by 255 to make the input value between 0 and 1. To match the above specific embodiment, P = 4.

[0087] Step 3, send the processed pictures in sequence to the encoding module, and output the recognition result from the decoding module after encoding and attention mechanism.

[0088] Step 4, display the recognition result in any form set, such as text, table, etc.

[0089] Accordingly, a picture recognition device can be further provided, comprising:

[0090] a picture collecting part for collecting pictures to be recognized. In hardware, it can be a color camera or an event camera and the like hardware device.

[0091] a picture preprocessing part for normalizing the collected pictures and copying them P times to form a picture sequence. The picture preprocessing part can be an integrated software module.

[0092] a processing and storage part for storing and running the picture recognition system of the present application, taking the picture sequence obtained by the picture preprocessing part as input to obtain a recognition result.

[0093] a display part for displaying the recognition result together with the original picture. For example, it can display the final effect in the form of labeling the classification result on the original picture in text.

[0094] The picture preprocessing part and the processing and storage part can be integrated.

[0095] The above is only a preferred embodiment of the present application, and the present application should not be limited to the content disclosed in the embodiment and the drawings. Any equivalent or modification made without departing from the disclosed spirit falls within the protection scope of the present application.

Claims

1. A picture recognition system based on a pulse neural network with a self-attention mechanism, characterized in that, include: The encoding module transforms the input image sequence into a pulse sequence through a convolutional coding layer, and then adds learnable positional coding to obtain an image pulse sequence with spatiotemporal information. The attention module consists of several layers of multi-head self-attention blocks stacked together. Each layer of multi-head self-attention blocks comprises four modules: layer normalization, temporal multi-head attention mechanism, spatial multi-head attention mechanism, and multilayer perceptron. The layer normalization normalizes the image pulse sequence with spatiotemporal information to stabilize the network's learning process. The temporal multi-head attention mechanism extracts features in the temporal dimension, the spatial multi-head attention mechanism extracts features in the spatial dimension, and the multilayer perceptron adds nonlinear factors to the network, enhancing its representational ability. The decoding module receives the pulse sequence processed by the attention module, obtains the final feature vector through sequence pooling, and finally sends it to the fully connected head for image recognition. In this context, the spiking neurons in the convolutional coding layer and attention module use membrane potential to represent their state and use ordinary differential equations to update their state. The equation parameters are intra-neural parameters with nonlinear time-varying characteristics. The state of the spiking neuron is updated jointly by the current state and the influence of the input synapse. The real-time membrane potential of the spiking neuron itself tends to recover towards the resting potential over time. A simulated gradient function is used to replace the position of the activation function of the spiking neuron in the computational graph during backpropagation. The formula for the simulated gradient function is: The derivative is: α is a hyperparameter that is manually specified and used to control the slope of the simulated gradient function.

2. The picture recognition system based on the pulse neural network with self-attention mechanism according to claim 1, characterized in that, The encoding module includes a convolutional encoding layer and positional encoding; The convolutional coding layer is a convolutional neural network designed for spiking neural networks, used to transform the input image sequence into a pulse sequence; The position encoding is used to add spatiotemporal relative order information to the pulse sequence.

3. The picture recognition system based on the pulse neural network with self-attention mechanism according to claim 1, characterized in that, The mathematical expression of the state equation corresponding to the spiking neuron is as follows: Where H[t] and V[t] correspond to the membrane potentials of the spiking neuron after receiving input and after the trigger pulse at time steps t = 1, 2, ..., T, respectively. When t = 0, the neuron state is initialized to 0. X[t] is the input received from the previous layer network at time step t, which is transformed into a pulse sequence after being processed by the spiking neuron. τ is the membrane time constant, defined as follows: C m denotes the membrane capacitance of the spiking neuron, g l is the membrane conductance of the spiking neuron; The formula for the trigger pulse is as follows: S[t] = Θ(H[t] - V th ) S[t] is the output pulse at time step t, Θ(x) is the step function, Θ(x) = 1 when x > 0, otherwise Θ(x) = 0, V th is the threshold for the firing pulse; The formula for obtaining V[t] after updating H[t] after the trigger pulse is: V[t] = H[t] (1 - S[t]) + V reset S[t] V reset is a reset potential following the trigger pulse.

4. The picture recognition system based on a pulse neural network with self-attention mechanism according to claim 1, characterized in that, Both the temporal multi-head attention mechanism and the spatial multi-head attention mechanism are self-attention operations, which operate in the temporal and spatial dimensions, respectively. The operational formula for the spatial self-attention mechanism is as follows: The operational formula for the temporal self-attention mechanism is as follows: l = 1,..., L, represents the layer number to which the multi-head self-attention block belongs, h = 1,..., H, represents the head to which the multi-head self-attention belongs, p and p' represent the position of the vector in the spatial dimension, the value range is {1,..., N}, t and t' represent the position of the vector in the time dimension, the value range is {1,..., T}, L, H, N, T respectively represent the total number of layers of the multi-head self-attention block, the total number of heads of the multi-head self-attention block, the total spatial and time dimensions of the multi-head self-attention block; and is the vector of the vector at the spatial position p and the time position t of the hth head of the lth layer multi-head self-attention block; is the vector of the vector at the spatial position p and the time position t of the hth head of the lth layer multi-head self-attention block; is the vector of the vector at the spatial position p and the time position t of the hth head of the lth layer multi-head self-attention block; given the output z after position encoding processing, k / q / v is obtained by linear transformation on z, SM(x) is a normalized exponential function, which is used to normalize the given vector x into a probability distribution with a sum of 1, D h = D / H is the dimension of each head, where D is the dimension of the input vector; Finally, the self-attention operation concatenates each head and performs a linear mapping to obtain the final result: s is the output vector with more rich spatiotemporal representation after time or spatial multi-head self-attention processing, Concat(x,...,y) is a concatenation function, W O is a linear mapping matrix; The multilayer perceptron consists of fully connected layers and spiking neurons, as formulated as follows: z (l) = FC(SN(FC(s (l) ))) s (l) is the result of the time and space multi-head self-attention mechanism processing in turn for the input of the first layer multi-head self-attention block, z (l) is the final output of the first layer multi-head self-attention block, FC represents a fully connected layer, and SN represents a spiking neuron.

5. The picture recognition system based on a pulse neural network with self-attention mechanism according to claim 1, characterized in that, The decoding module includes a sequence pooling layer and a fully connected classification head; The sequence pooling layer actively learns and assigns different weights to features of different sequence segments, as formulated as follows: o = SM(FC(z (L) )) T z (L) wherein L represents z (L) The output of the last multi-head self-attention block is o, which is finally input into a fully connected classification head to obtain the final network picture recognition result.

6. A picture recognition method using the picture recognition system based on the pulse neural network with self-attention mechanism according to claim 1, comprising the following steps: capturing pictures by using a camera to obtain any required number of pictures; normalizing the pictures and copying each picture P times to form a picture sequence; sending the processed pictures in sequence to the encoding module, and outputting a recognition result from the decoding module through encoding and attention mechanism; displaying the recognition result.

7. A picture recognition device, comprising: a picture capturing part for capturing pictures to be recognized; a picture preprocessing part for normalizing the captured pictures and copying them P times to form a picture sequence; a processing and storage part for storing and running the picture recognition system according to claim 1, taking the picture sequence obtained by the picture preprocessing part as input, and obtaining a recognition result; a display part for displaying the recognition result together with the original picture.

8. The picture recognition device according to claim 7, characterized in that The picture capturing part is a color camera or an event camera; the display part displays the classification result in the form of text marked on the original picture.

Images