An optimization method for YOLO spiking neural network
By analyzing, standardizing and converting the YOLO artificial neural network, the YOLO pulse neural network is constructed, which solves the problem of low accuracy of YOLO pulse neural network in the existing technology, improves image recognition accuracy and achieves higher energy-efficient calculations.
Patent Information
- Application Number
- CN202011117592.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-19
- Publication Date
- 2025-03-18
- Estimated Expiration
- 2040-10-19
AI Technical Summary
In the prior art, the accuracy of YOLO pulse neural network is low and it is difficult to meet the strict requirements for detection frame coordinates and probability size in object detection.
By analyzing, standardizing and converting the YOLO artificial neural network, a YOLO pulse neural network is built to improve its accuracy. The specific steps include analyzing the network structure of the YOLO artificial neural network, performing dynamic standardization processing, and converting it into a YOLO pulsed neural network.
The image recognition accuracy of YOLO pulse neural network is improved, the goal of completing image classification tasks with a sparse network is achieved, and more energy-efficient calculations are achieved while ensuring the target accuracy.
Smart Images
Figure CN112232486B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of neural network, and more specifically, relates to an optimization method for a YOLO pulse neural network. Background Art
[0002] Object detection is a practical and challenging computer vision task, which can be seen as a combination of image classification and positioning. The object detection system must be able to identify the category of the image and give the precise location of the target. Therefore, object detection is more complex than classification tasks. Yolo, a neural network based on a deep learning framework, can provide end-to-end object detection services. The network divides the input image into M×N grids, and then each cell is responsible for detecting the targets whose center points fall within the grid.
[0003] The pulse neural network is the third generation of neural networks. Its simulated neurons are closer to reality. In addition, the influence of time information is also taken into account. The idea is that the neurons in the dynamic neural network are not activated in every iteration (as in the typical multi-layer perceptron network), but are activated only when its membrane potential reaches a certain value. When a neuron is activated, it generates a signal to other neurons, raising or lowering their membrane potential.
[0004] In a spiking neural network, the current activation level of a neuron (modeled as a differential equation) is usually considered to be the current state. An input pulse will increase the current value for a period of time, and then gradually decay. There are many encoding methods that interpret these output pulse sequences as an actual number. These encoding methods take into account both the pulse frequency and the pulse interval time. With the help of neuroscience research, people can accurately establish a neural network model based on the time of pulse generation. This new type of neural network uses spike coding. By obtaining the exact time when the pulse occurs, this new type of neural network can obtain more information and stronger computing power.
[0005] The current research status shows that the training of spiking neural networks is still in its infancy, and the research on the conversion from artificial neural networks to spiking neural networks is still limited to shallow neural networks, and most applications are concentrated on image classification that only requires comparison of relative probability sizes. However, since the target detection algorithm requires accurate detection frame coordinates and probability sizes, it has more stringent requirements on the output accuracy of the spiking neural network during the conversion process. Therefore, how to improve the accuracy of the YOLO spiking neural network is an urgent problem to be solved by existing technologies. Summary of the invention
[0006] 1. Problem to be solved
[0007] In view of the problem of low accuracy of YOLO pulse neural network in the prior art, the present invention provides an optimization method of YOLO pulse neural network. By parsing, standardizing and converting the YOLO artificial neural network, the obtained YOLO pulse neural network has high accuracy, thereby improving the image recognition accuracy of the YOLO pulse neural network.
[0008] 2. Technical solution
[0009] In order to solve the above problems, the technical solution adopted by the present invention is as follows:
[0010] The invention discloses an optimization method of a YOLO pulse neural network, comprising: constructing a YOLO artificial neural network, and parsing the network structure of the YOLO artificial neural network; performing dynamic standardization processing on the parsed YOLO artificial neural network to obtain a standardized YOLO artificial neural network; and then converting the standardized YOLO artificial neural network into a YOLO pulse neural network.
[0011] Furthermore, the specific process of parsing the network structure of the YOLO artificial neural network is as follows: extracting the properties and parameters of the convolutional layer, batch normalization layer and activation layer in the YOLO artificial neural network; the parameters of the convolutional layer linearly absorb the parameters of the batch normalization layer to obtain the parsing parameters of the convolutional layer; then detecting the connection relationship between the convolutional layer and the activation layer, and processing the properties of the convolutional layer according to the connection relationship to obtain the parsing properties of the convolutional layer; and then constructing the parsed YOLO artificial neural network according to the parsing parameters and parsing properties of the convolutional layer.
[0012] Furthermore, the specific process of processing the properties of the convolution layer according to the connection relationship is: if the activation layer is embedded in the convolution layer, the properties of the convolution layer will not be changed; if the activation layer is external to the convolution layer, the properties of the convolution layer linearly absorb the properties of the activation layer to obtain the analytical properties of the convolution layer.
[0013] Furthermore, the specific process of dynamically standardizing the parsed YOLO artificial neural network is as follows: samples of the training set of the YOLO artificial neural network are selected and input into the convolution layer; the convolution layer obtains the activation value distribution according to the samples, and determines the scaling factor of the convolution layer according to the activation value distribution; the convolution layer is standardized according to the scaling factor to obtain a standardized convolution layer, and then the standardized convolution layer, batch normalization layer and activation layer are combined to form a standardized YOLO artificial neural network.
[0014] Furthermore, the specific process of converting the standardized YOLO artificial neural network into the YOLO spiking neural network is as follows: a pulse input and output mechanism, a pulse emission mechanism and a membrane potential reset mechanism are established for the standardized YOLO artificial neural network to obtain the YOLO spiking neural network.
[0015] Furthermore, the pulse emission mechanism is established according to the activation function of the convolutional layer of the standardized YOLO artificial neural network.
[0016] Furthermore, when the convolutional layer of the standardized YOLO artificial neural network has no activation function, the pulse emission mechanism is established by the following formula:
[0017]
[0018] Here, x represents the input neuron and y represents the output activation value.
[0019] Furthermore, the specific process of establishing the input-output mechanism and the membrane potential reset mechanism is as follows: using the standardized YOLO artificial neural network to perform convolution operations on the pulses to obtain the membrane potential matrix, and then performing pulse input and membrane potential reset according to the membrane potential matrix.
[0020] 3. Beneficial effects
[0021] Compared with the prior art, the present invention has the following beneficial effects:
[0022] The invention discloses a YOLO spiking neural network optimization method, which can obtain an optimized YOLO spiking neural network by parsing, standardizing and converting the YOLO artificial neural network, and effectively improve the accuracy of the YOLO spiking neural network, thereby improving the image recognition accuracy of the YOLO spiking neural network and achieving the goal of completing the image classification task with a sparser network. Furthermore, under the premise of ensuring the target accuracy, the YOLO spiking neural network can utilize its large sparsity and no multiplication characteristics to achieve more energy-efficient computing. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0024] Figure 2 Schematic diagram of the conversion process of the maximum pooling layer;
[0025] Figure 3 A schematic diagram showing the correlation between the output value of the maximum pooling layer of the YOLO artificial neural network and the pulse firing rate of the corresponding layer of the YOLO spiking neural network;
[0026] Figure 4 This is a schematic diagram of the accuracy change of TINY YOLOv2 spiking neural network at different time lengths on the PASCAL dataset;
[0027] Figure 5 Schematic diagram of the comparison of the effects of pulse neural network and artificial neural network on target detection in images at different time lengths. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments; moreover, the various embodiments are not relatively independent and can be combined with each other as needed to achieve better results. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0029] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.
[0030] Example 1
[0031] Combination Figure 1 As shown, a YOLO pulse neural network optimization method of the present invention comprises the following steps:
[0032] 1) Analysis
[0033] Construct a YOLO artificial neural network and analyze the network structure of the YOLO artificial neural network; specifically, extract the properties and parameters of the convolution layer, batch normalization layer, and activation layer in the YOLO artificial neural network; the parameters include weights and biases; the parameters of the convolution layer linearly absorb the parameters of the batch normalization layer to obtain the analytical parameters of the convolution layer; then detect the connection relationship between the convolution layer and the activation layer, and process the properties of the convolution layer according to the connection relationship to obtain the analytical properties of the convolution layer. The properties include convolution kernel size, step size, activation mode, etc.
[0034] It is worth noting that if the activation layer is embedded in the convolution layer, the properties of the convolution layer will not be changed; if the activation layer is external to the convolution layer, the properties of the convolution layer linearly absorb the properties of the activation layer to obtain the parsing properties of the convolution layer. Furthermore, the parsed YOLO artificial neural network is constructed based on the parsing parameters and parsing properties of the convolution layer.
[0035] 2) Dynamic Standardization
[0036] The parsed YOLO artificial neural network is dynamically standardized to obtain a standardized YOLO artificial neural network; specifically, the samples of the training set of the YOLO artificial neural network are first selected and input into the convolution layer; the convolution layer calculates the activation value distribution according to the samples, and then determines the scaling factor of the convolution layer according to the activation value distribution; it is worth noting that the scaling factor is 99.9%-100% of the maximum activation value. Further, the activation value is tested for outliers. If the outliers are few and scattered, some neurons can be discarded to increase the neuron pulse firing rate; if the outliers are few and relatively concentrated, some neurons are retained, and then the convolution layer is standardized according to the scaling factor to obtain a standardized convolution layer, and then the standardized convolution layer, batch normalization layer and activation layer are combined to form a standardized YOLO artificial neural network.
[0037] It is worth noting that the scaling factor is used to standardize the parameters of the artificial neural network, reducing or enlarging the parameters to the range of parameters required by the spiking neural network, so that the spiking neural network can obtain the standardized neuron membrane potential value and perform normal neuron firing. By dynamically standardizing the parsed YOLO artificial neural network, the firing rate of the neuron can be increased without overflowing the firing rate.
[0038] 3) Conversion
[0039] The standardized YOLO artificial neural network is converted to a YOLO spike neural network. The specific process of the conversion is as follows:
[0040] The pulse input and output mechanism, pulse emission mechanism and membrane potential reset mechanism are established for the standardized YOLO artificial neural network to obtain the YOLO pulse neural network. It is worth noting that the pulse emission mechanism is established according to the activation function of the convolution layer of the standardized YOLO artificial neural network. The activation function only exists in the artificial neural network, but the activation function can be approximated as the pulse emission rate of the intergrate-and-fire neuron model. Furthermore, when the convolution layer of the standardized YOLO artificial neural network does not have the activation function of ReLu or LeakyReLu, there is a problem that it cannot be converted to the pulse emission mechanism, but it can be assumed that the last layer has the activation function of LeakyReLu with the extreme case a=1, and the pulse emission mechanism is established by the following formula:
[0041]
[0042] Among them, x represents the input neuron and y represents the output activation value. It is worth noting that the activation function can be simulated according to the characteristics of the pulse emission mechanism: when the neuron membrane potential exceeds the threshold, a pulse is emitted, and then the membrane potential is reset to prepare for the next pulse emission. When x>0, the threshold is set to 1 to simulate a positive pulse; when x<0, the threshold is set to -1 to simulate a negative pulse. The neuron value in the artificial neural network is simulated by the pulse emission rate of each neuron in a time window.
[0043] Furthermore, the specific process of establishing the input-output mechanism and the membrane potential reset mechanism is as follows: using the standardized YOLO artificial neural network to perform convolution operations on the pulses to obtain a membrane potential matrix, and then performing pulse input and membrane potential reset according to the membrane potential matrix. Specifically, the pulse consists of 1 or 0, and the pulse is used as the input value and convolved with the parameters of the convolution layer, batch normalization layer or activation layer. The result matrix obtained by the convolution operation is the membrane potential matrix.
[0044] Pulse emission is determined by whether the value in the membrane potential matrix exceeds a certain set emission threshold. If the membrane potential at a certain position exceeds the threshold, the pulse output at that position is 1. If the membrane potential does not exceed the threshold at that position, the pulse output at that position is 0, that is, no pulse is emitted. In addition, if a pulse is emitted at a certain position, the membrane potential at that position is reset to 0.
[0045] In addition, for the maximum pooling layer in the YOLO artificial neural network, if the maximum pooling operation is directly performed on the input pulse matrix, the neuron with the largest pulse firing rate cannot be correctly determined. Therefore, it is necessary to record the accumulated value of the non-reset membrane potential of the layer above the maximum pooling layer, and use the accumulated value of the membrane potential and the pulse as the input of the maximum pooling layer to jointly determine the pulse output of this layer. By converting the maximum pooling layer, the accuracy of the YOLO pulse neural network is improved. Figure 2 As shown in the figure, firstly, a maximum pooling index operation is performed on the neuron membrane potential matrix to obtain the index of the maximum neuron membrane potential in each window, and then the value of the corresponding input neuron pulse is found according to the index to obtain the output pulse of the maximum pooling layer. Figure 3 This is a comparison chart of the correlation between the output value of the maximum pooling layer of the YOLO artificial neural network and the pulse firing rate of the neurons in the spiking neural network.
[0046] The invention discloses a YOLO spiking neural network optimization method, which can obtain an optimized YOLO spiking neural network by parsing, standardizing and converting the YOLO artificial neural network, and effectively improve the accuracy of the YOLO spiking neural network, thereby improving the image recognition accuracy of the YOLO spiking neural network and achieving the goal of completing the image classification task with a sparser network. Furthermore, under the premise of ensuring the target accuracy, the YOLO spiking neural network can utilize its large sparsity and no multiplication characteristics to achieve more energy-efficient computing.
[0047] In this embodiment, the above-mentioned YOLO spiking neural network optimization method is used to conduct experiments on a YOLO model TINYYOLOv2 for the PASCAL dataset. Figure 4 As shown, the experimental results show that the accuracy of TINYYOLOv2 on the PASCAL dataset stabilizes at 50.70% after 3500 time lengths, with a precision loss of about 1.7%; the method of the present invention is used to experiment on a YOLOv2 network of the YOLO model for the COCO dataset. The experimental results show that the accuracy of YOLOv2 on the COCO dataset stabilizes at 45.30% after 3000 time lengths, with a precision loss of about 3%. The experimental results also prove that dynamic normalization (which can improve by 3% compared to no optimization) and the maximum pooling method of membrane potential cumulative values (which can improve by 15% compared to no optimization) can improve accuracy. In addition, Figure 5 The results of target detection on images by spiking neural networks at different time lengths are compared with those by artificial neural networks, proving the effectiveness of the converted YOLO spiking neural network in target detection tasks.
[0048] The present invention has been described in detail above in conjunction with specific exemplary embodiments. However, it should be understood that various modifications and variations may be made without departing from the scope of the present invention as defined by the appended claims. The detailed description and the accompanying drawings should be considered only as illustrative and not restrictive, and if there are any such modifications and variations, they will fall within the scope of the present invention described herein. In addition, the background technology is intended to illustrate the current status and significance of the research and development of the present technology and is not intended to limit the present invention or the application field of the present application and the present invention.
Claims
1. A YOLO spiking neural network optimization method, characterized in that: include: Construct the YOLO artificial neural network and analyze the network structure of the YOLO artificial neural network; The parsed YOLO artificial neural network is dynamically normalized to obtain a standardized YOLO artificial neural network; Convert the normalized YOLO artificial neural network to a YOLO spiking neural network; Use YOLO spiking neural network to detect image targets and improve image recognition accuracy; The specific process of parsing the network structure of the YOLO artificial neural network is as follows: Extract the properties and parameters of the convolutional layer, batch normalization layer, and activation layer in the YOLO artificial neural network; The parameters of the convolutional layer linearly absorb the parameters of the batch normalization layer to obtain the analytical parameters of the convolutional layer; Detect the connection relationship between the convolution layer and the activation layer, and process the properties of the convolution layer according to the connection relationship to obtain the analytical properties of the convolution layer; Construct the parsed YOLO artificial neural network based on the parsing parameters and parsing properties of the convolutional layer; The specific process of processing the attributes of the convolutional layer according to the connection relationship is: If the activation layer is embedded in the convolutional layer, the properties of the convolutional layer will not be changed; If the activation layer is external to the convolutional layer, the properties of the convolutional layer linearly absorb the properties of the activation layer to obtain the analytical properties of the convolutional layer; The specific process of dynamic normalization of the parsed YOLO artificial neural network is as follows: Select samples from the training set of the YOLO artificial neural network and input them into the convolutional layer; The convolution layer calculates the activation value distribution based on the samples and determines the scaling factor of the convolution layer based on the activation value distribution; The convolution layer is normalized according to the scaling factor to obtain a standardized convolution layer, and then the standardized convolution layer, batch normalization layer and activation layer are combined to form a standardized YOLO artificial neural network; The specific process of converting the standardized YOLO artificial neural network into the YOLO spiking neural network is as follows: establishing a pulse input and output mechanism, a pulse emission mechanism and a membrane potential reset mechanism for the standardized YOLO artificial neural network to obtain the YOLO spiking neural network.
2. The optimization method of a YOLO spiking neural network according to claim 1, wherein: The pulse emission mechanism is established according to the activation function of the convolutional layer of the standardized YOLO artificial neural network.
3. The optimization method of a YOLO spiking neural network according to claim 1, wherein: When the convolutional layer of the standardized YOLO artificial neural network has no activation function, the pulse emission mechanism is established by the following formula: Here, x represents the input neuron and y represents the output activation value.
4. The optimization method of a YOLO spiking neural network according to claim 1, wherein: The specific process of establishing the input-output mechanism and the membrane potential reset mechanism is: using the standardized YOLO artificial neural network to perform convolution operations on the pulses to obtain the membrane potential matrix, and then performing pulse input and membrane potential reset according to the membrane potential matrix.