A pulse neural network target detection method based on uniform poisson coding

Abstract

The application provides a pulse neural network target detection method based on uniform Poisson coding, constructs a pulse neural network model to realize a video target detection algorithm, and uses the low power consumption characteristics of the pulse neural network on a dedicated hardware to provide a software scheme for edge device application of the target detection algorithm. The application takes YOLOv4-tiny as a model basis and uses an ANN-SNN training method. Poisson coding is used for image preprocessing in ANN training, reduces information loss caused by using Poisson coding during prediction, and increases the robustness of the system and the recognition of occluded objects. The original Leak-Relu function in the ANN model and the IF neuron in the SNN model are improved, further reducing the error caused by ANN-SNN, and improving the prediction accuracy of the SNN.

Description

TECHNICAL FIELD

[0001] The application relates to the field of brain-like intelligence and computer vision, and is a target detection method of a pulse neural network based on uniform Poisson coding. BACKGROUND

[0002] Artificial neurons were first proposed in the 1940s and 1950s, and the purpose was to simulate biological neurons and brain functions to achieve intelligent learning. With the rapid development of computer hardware and software technology in the 21st century, artificial neural networks have gradually become an effective computing model, and important achievements have been made in the fields of image recognition, natural language processing, robot control, medical detection and the like. However, with the continuous expansion of the application model, the power consumption and memory required by the model have become a bottleneck restricting its development. For example, AlphaGo needs 1202 CPUs and 176 GPUs to predict a game of Go, which is a great challenge for application in edge devices and scenarios with limited storage space.

[0003] Under this background, the pulse neural network (SNN) has attracted widespread attention. As one of the important research fields of brain-like intelligence, the pulse neural network is significantly different from the artificial neural network in essence. The neurons in the pulse neural network simulate the neurons in the human brain, and do not directly emit signals after receiving input, but only produce a discrete pulse signal when the cumulative input is greater than or equal to the threshold, which is transmitted to the next neuron through the synapse. The generation and transmission process of such a discrete signal is more biologically interpretable. At the same time, since the output has a discrete feature, it can run on event-driven hardware. Compared with traditional clock-driven processors, event-driven hardware only performs corresponding operations when an event occurs, and has a significant advantage in power consumption. For example, the TrueNorth chip of IBM has an energy consumption of only 3-4 orders of magnitude of that of a traditional microprocessor. Although the pulse neural network has the characteristics of low power consumption and high biological interpretability, it is difficult to train, which limits its development.

[0004] Target detection is an important computer vision technology, which aims to improve the intelligent level of computer vision systems by analyzing and recognizing targets in images or videos. With the development of artificial neural network technology, target detection technology is increasingly widely used in intelligent driving, medical care and other fields. For example, in the field of intelligent driving, target detection technology can help the car system to identify the surrounding vehicles and pedestrians, and improve the driving safety. In the medical field, target detection technology can help doctors quickly locate the site of the disease, and assist doctors in accurately diagnosing the disease. However, traditional target detection methods usually require a large amount of power consumption and storage space, which is difficult to apply to edge devices, which may not be suitable for some resource-constrained scenarios. Therefore, it is necessary to explore more lightweight and efficient target detection methods to meet the requirements of edge devices.

[0005] The low power consumption of spiking neural network on brain-like hardware can solve the problem that traditional target detection is difficult to apply to edge devices due to power consumption and space. It is a relatively new attempt in both the field of spiking neural network and the field of target detection. In 2020, Kim et al. made the first attempt of spiking-YOLO, which proved the feasibility of this direction. However, due to the error of ANN-SNN training method, it needs more than 3000 time steps to have better performance. In addition, most SNNs will produce some errors due to Poisson coding, so they choose to input directly based on images, which brings great challenges to the hardware. SUMMARY

[0006] The purpose of the present application is to provide a Poisson coding-based spiking neural network target detection method, to construct a spiking neural network model to realize a video target detection algorithm, and to utilize the low power consumption of spiking neural network on dedicated hardware to provide a software solution for the edge device application of target detection algorithm.

[0007] To achieve this purpose, the present application adopts the following technical solutions:

[0008] A Poisson coding-based spiking neural network target detection method, comprising the following steps:

[0009] Step S1, selecting YOLOv4-tiny network as ANN model. In the image preprocessing stage, the image is encoded using multi-channel Poisson coding, which generates a total of 512 time steps of Poisson coding. The encoded picture is subjected to data enhancement in three dimensions: superimposing the original encoding, discarding or filling random areas within the entire time step range, and discarding or filling random event areas within the random time step range. The ANN model is pre-adjusted, and the max pooling is replaced by average pooling.

[0010] Step S2, training the ANN model to obtain the weights.

[0011] Step S3, using the ANN model for prediction to obtain the output of each activation layer neuron of the ANN. A total of 5000 different pictures are predicted, and the positive and negative activation values of the 99.9% peak position in the 5000 outputs of each neuron are selected as the threshold of the subsequent SNN and the threshold of the Leak-ReLU activation function in the ANN.

[0012] Step S4, using the Leak-ReLU activation function with threshold to continue training the ANN model on the original weights to obtain the latest ANN weights. The Leak-ReLU activation function is described as:

[0013]

[0014] where x represents the input, a represents the slope of the Leak-ReLU activation function in the negative activation part, represents the 99.9% peak position of the positive activation in the ANN, represents the 99.9% peak position of the negative activation in the ANN.

[0015] Step S5, using soft reset to modify the IF neuron in the SNN as:

[0016]

[0017]

[0018] v l (t)=v l (t-1)+W l x l-1 (t)+b l -x l (t) (4)

[0019] In the formula, the vector represents the threshold, the vector W l , b l represents the weight and bias of the lth layer, the vector represents the membrane potential of the lth layer neuron before firing at time t, the vector v l (t-1) represents the residual membrane potential of all neurons in the lth layer at time t-1, and the vector x l-1(t) represents the pulse released by the l-1 layer neuron at time t, and is also the input of the l layer; for the hard reset neuron, the membrane potential will immediately return to the resting potential after emitting a pulse. Therefore, the hard reset neuron ignores the residual potential at the moment of discharge, resulting in a decrease in the accuracy of the converted SNN. Soft reset neurons avoid the above problems and are widely used in various models for classification tasks, such as VGG16, ResNet-20.

[0020] Add negative threshold and negative activation. Traditional IF neurons can only deal with positive activation, while more complex networks often use Leak-ReLU activation functions. If traditional IF neurons are used, negative activation will be ignored, and the prediction effect will be greatly reduced. The improved IF neuron adds negative activation and negative threshold, and at the same time reduces the error brought by ANN-SNN method by setting the initial membrane voltage and neuron bias. Finally, the membrane potential v of the improved neuron is l (t) is described as:

[0021]

[0022]

[0023]

[0024] Set the initial threshold When the neuron input is positive When the input is negative

[0025] Step S6, build an SNN target detection model, select to use multi-channel Poisson coding to change the picture into 0, 1 pulse, and distribute it uniformly as input, replace the Leak-ReLU function in the ANN model with the improved IF neuron, that is, the target detection model based on the pulse neural network can be completed.

[0026] The present application uses the ANN-SNN training method based on the YOLOv4-tiny model. Poisson coding is used in ANN training to preprocess the image, which reduces the information loss caused by using Poisson coding during prediction and increases the robustness of the system and the recognition of occluded objects. The original Leak-ReLU function in the ANN model and the IF neuron in the SNN model are improved, further reducing the error caused by ANN-SNN and improving the prediction accuracy of SNN. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a schematic diagram of the YOLOV4-tiny model structure based on ANN;

[0028] Figure 2is a schematic diagram of a YOLOV4-tiny model structure based on SNN;

[0029] Figure 3 is a schematic diagram of the original YOLOv4-tiny prediction result;

[0030] Figure 4 is a schematic diagram of the YOLOV4-tiny prediction result based on SNN;

[0031] Figure 5 is a specific flowchart of target detection by a spiking neural network

[0032] Figure 6 is a pre-Poisson encoding image

[0033] Figure 7 is a schematic diagram at different time steps after Poisson encoding

[0034] Figure 8 is a conversion error schematic diagram

[0035] Figure 9 is a translation error conversion error schematic diagram DETAILED DESCRIPTION

[0036] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope of the appended claims of the application.

[0037] The application selects the YOLOV4-tiny classic target detection network model as the basis, adds Poisson encoding data enhancement in the ANN training stage, and improves the traditional Leak-ReLU activation function, trains the weight and saves it. The SNN model is adjusted on the basis of YOLOV4-tiny, uses uniform Poisson encoding as input, replaces the Leak-ReLU activation function in ANN with the improved IF neuron, and replaces MaxPool with AvgPool, and finally the target detection model structure based on SNN is as shown in Figure 2 The weight obtained by ANN training is applied to the SNN model, and finally the target detection using the spiking neural network is realized. The specific process is shown in Figure 5 .

[0038] A spiking neural network target detection method based on uniform Poisson encoding, comprising the following steps:

[0039] Step S1, select YOLOv4-tiny network as ANN model. In the image preprocessing stage, use multi-channel Poisson encoding to encode the image, a total of 512 time steps of Poisson encoding are generated. The picture after Poisson encoding is subjected to data enhancement in three dimensions: superimposition of original encoding, discarding or filling of random areas in the entire time step range, and discarding or filling of random event areas in random time steps. The ANN model is pre-adjusted, and the max-pooling is replaced by average-pooling.

[0040] Step S2, train the ANN model to obtain the weight.

[0041] Step S3, use the ANN model to make prediction, and obtain the output of each activation layer neuron of the ANN. A total of 5000 different pictures are predicted, and the positive and negative activation values of the 99.9% peak position in 5000 outputs of each neuron are selected as the threshold of the subsequent SNN and the threshold of the Leak-ReLU activation function in the ANN.

[0042] Step S4, use the Leak-ReLU activation function with threshold to continue training the ANN model on the original weight, and obtain the latest ANN weight. The Leak-ReLU activation function is described as:

[0043]

[0044] where x represents the input, a represents the slope of the Leak-ReLU activation function in the negative activation part, represents the 99.9% peak position of the positive activation in the ANN, represents the 99.9% peak position of the negative activation in the ANN.

[0045] Step S5, use soft reset to change the IF neuron in the SNN to:

[0046]

[0047]

[0048] v l (t)=v l (t-1)+W l x l-1 (t)+b l -x l (t) (4)

[0049] In the formula, the vector represents the threshold, the vector W l , b l represents the weight and bias of the lth layer, and the vector represents the membrane potential of the l-th neuron before firing at time t, vector v l (t-1) represents the residual membrane potential of all neurons in the l-th layer at time t-1, vector x l-1 (t) represents the pulse released by the l-1-th layer neuron at time t, and is also the input of the l-th layer. For hard-reset neurons, the membrane potential will immediately return to the resting potential after firing a pulse. Therefore, hard-reset neurons ignore the residual potential at the moment of firing, resulting in a decrease in the accuracy of the converted SNN. Soft-reset neurons avoid the above problems and are widely used in various models for classification tasks, such as VGG16 and ResNet-20.

[0050] Add negative threshold and negative activation. Traditional IF neurons can only deal with positive activation, while more complex networks often use Leak-ReLU activation functions. If traditional IF neurons are used, negative activation will be ignored, and the prediction effect will be greatly reduced. The improved IF neuron adds negative activation and negative threshold, and at the same time sets the initial membrane voltage and neuron bias to reduce the error caused by ANN-SNN method. The final improved neuron membrane potential v l (t) is described as:

[0051]

[0052]

[0053]

[0054] Set the initial threshold When the neuron input is positive When the input is negative

[0055] Step S6, build an SNN target detection model, select the multi-channel Poisson coding to change the picture into 0,1 pulse as input, replace the Leak-ReLU function in the ANN model with the improved IF neuron, that is, complete the construction of the target detection model based on the pulse neural network.

[0056] The YOLOv4-tiny network structure is divided into three parts: backbone network part, feature fusion part, and YOLO Head part, as shown in Figure 1 .

[0057] In the aspect of dataset selection, VOC2007+VOC2012 dataset is selected, in which the training dataset has a total of 19353 images, and the test dataset has a total of 2140 images. The dataset contains 20 common object categories, such as people, cars, airplanes, dogs, cats, etc., and provides a large number of images and corresponding object annotation information, including the bounding box and category of each object. The resolution size of each image varies from 100*100 to 500*500.

[0058] The dataset encoding is selected as Poisson encoding. The basic idea of Poisson encoding is to calculate the probability of each pixel point by modeling the probability distribution of the image, and to encode the pixel points with high probability as short codes and the pixel points with low probability as long codes. Poisson encoding not only considers the probability distribution of the pixel points, but also considers the relationship between the pixel points and the adjacent pixel points, so that the encoding result is more compact. Poisson encoding is a lossless compression image encoding method. Poisson encoding obeys the Poisson distribution probability density formula:

[0059]

[0060] In the formula, x is the number of events occurring in a fixed time or space, n is the specific number, λ is the average number of events occurring in a unit time or unit space, and e is the base number of natural logarithm.

[0061] When an image is input, dividing each pixel point in the image by 255 will obtain the probability of the pixel point emitting a pulse at each time step. After Poisson encoding, each pixel point at each time step will become a 0, 1 pulse. By setting the time step to 255, superimposing the pulses at each time step, and finally normalizing, the original image is obtained. Figure 6 The original image is Figure 7 The new images obtained after superimposing the time steps are 1, 100, 150, 200, 255, respectively. Experimental results show that the similarity between the new image obtained after Poisson encoding and the original image will be more than 99% with the increase of the time step.

[0062] Using the pulse matrix of 255 time steps, random discarding or increasing pulses in the random time step dimension, discarding or increasing pulses in a certain region of the pulse matrix in all time steps, or superimposing the original pulses to obtain the encoded image for data enhancement.

[0063] After fine-tuning the ANN model, the ANN is trained. The prior box size is obtained by the kmeans clustering algorithm, the input image size is [416, 416], the initial learning rate is 0.01, the minimum learning rate is 0.0001, and the stochastic gradient descent (SGD) optimizer is used for parameter optimization.

[0064] After training, the weights are saved for subsequent SNN models. Using the trained ANN model for prediction, 5000 different pictures are selected for prediction, and the output of each activated neuron is recorded. The positive activation or negative activation of 99.9% of each neuron peak is selected, and the positive threshold and negative threshold of Leak-ReLU are set. The ANN model with the modified activation function is further trained on the original weights to obtain the final weights.

[0065] The IF neuron is improved by adding a negative threshold and negative activation. The improved IF neuron firing formula is:

[0066]

[0067] The membrane voltage of the neuron at time t is:

[0068]

[0069] The membrane voltage of the lth layer neuron is accumulated at T time steps to obtain:

[0070]

[0071] Let the vector represent the average output of the l-1th layer neuron, and the initial membrane potential is set to 0. The average output z of the lth layer can be obtained l :

[0072]

[0073] According to the characteristics of neuron firing pulses, only when the membrane potential is greater than or equal to the positive threshold or the membrane potential is less than or equal to the negative threshold, the pulse will be fired, v l (T) is the remaining membrane potential that does not affect pulse firing, and the final average output can be represented as:

[0074]

[0075] Comparing the improved Leak-ReLU activation function: the error of the two is obtained, and the positive output part negative output part

[0076]

[0077] Set the initial threshold When the neuron input is positive When the input is negative At this time, the output error of the IF neuron and Leak-ReLU is minimized. Among them Figure 8 represents the error before improvement, Figure 9denotes the improved error.

[0078] Replace the activation function Leak-ReLU with the improved IF neuron, and use Poisson coding for image input. The coded spikes are uniformly distributed in time steps. Thus, the target detection system based on spiking neural network is built.

[0079] Experimental results analysis:

[0080] The mAP value predicted by ANN is shown in Figure 3 The prediction time step of SNN target detection is shortened from 2048 to 512 under the same average precision map, greatly reducing the prediction time, and the mAP is improved from 0.512 to 0.591. The feasibility and universality of this method are also proved in the classification task. Take the ResNet-20 model on the CIFAR18 dataset as an example, the accuracy rate reaches 94.5% at 32 time steps, compared with 94.3% at 256 time steps before. Poisson coding data augmentation is added to the training in ANN, and the map remains at 0.582 at time step 512, and the mAP reaches 0.639 at time step 1024, as shown in Figure 4 .

[0081] In summary, by improving the IF neuron, Leak-ReLU activation function, and adding Poisson coding data augmentation to the training in ANN, the average precision of SNN target detection can be effectively improved, and the time step required for prediction can be reduced. Compared with Spiking-YOLO without using Poisson coding, the time step required for the same average precision is reduced from 3000 to 256, and the average precision map is improved from 0.43 to 0.591 at the same time step 512. After using Poisson coding, the prediction ability is not greatly reduced due to coding, and the average precision map remains at 0.582.

Claims

1.A method for object detection based on uniform Poisson coding pulse neural network, characterized in that The method comprises the following steps: Step S1, selecting YOLOv4-tiny network as ANN model, using multi-channel Poisson encoding to encode the image in the image preprocessing stage, and a total of 512 time steps of Poisson encoding are generated; the picture after Poisson encoding is subjected to data enhancement according to three dimensions: superposition of original encoding, discarding or filling in a random area in the whole time step range, and discarding or filling in a random event area in a random time step; the ANN model is pre-adjusted, and the maximum pooling is replaced by average pooling; Step S2, training the ANN model to obtain the weight; Step S3, using the ANN model for prediction to obtain the output of each activation layer neuron of the ANN; a total of 5000 different pictures are predicted, and the positive activation value and the negative activation value of the 99.9% peak position in the 5000 outputs of each neuron are selected as the threshold of the subsequent SNN and the threshold of the Leak-ReLU activation function in the ANN; Step S4, using the Leak-ReLU activation function with threshold to continue training the ANN model on the original weight to obtain new ANN weight; the Leak-ReLU activation function is described as: where x denotes the input, a denotes the slope of the Leaky-ReLU activation function at the negative activation part, denotes the positive activation of the 99.9% peak in the ANN, denotes the negative activation of the 99.9% peak in the ANN; Step S5, using soft reset to change the IF neuron in the SNN to: v l (t) = v l (t - 1) + W l x l-1 (t) + b l - x l (t) (4) where vector denotes the threshold value, vector W l , b l denotes the weight and bias of the lth layer, vector denotes the membrane potential of the lth layer neuron before firing at time t, vector v l (t-1) denotes the residual membrane potential of all neurons in the lth layer at time t-1, vector x l-1 (t) denotes the pulse released by the l-1th layer neuron at time t, which is also the input of the lth layer; Add a negative threshold and a negative activation; The improved IF neuron adds negative activation and negative threshold, and at the same time reduces the error brought by the ANN-SNN method by setting the initial membrane voltage and neuron bias. The final improved neuron membrane potential v l (t) is described as: Setting an initial threshold When the neuron input is positive When the input is negative Step S6, build SNN target detection model, select multi-channel Poisson coding to change the picture into 0, 1 pulse, make it uniform distribution as input, replace the Leak-ReLU function in ANN model with improved IF neuron, complete the target detection model based on pulse neural network.

Images