A low-power target detection method based on convolutional spiking neural network

Abstract

The present invention discloses a low-power target detection method based on a convolutional pulse neural network, including: establishing a weed picture data set and performing preprocessing; constructing a pulse neural network model for weed detection according to the deep convolutional neural network conversion conditions; performing neuron information transmission on the pulse sequence of the pulse neural network model, and obtaining the leakage integral emission model neuron accumulation input into the membrane voltage; using the weed picture data set to train the pulse neural network model, realizing its fine-grained normalization, and setting model parameters; inputting the weed picture to be identified into the trained pulse neural network model to obtain a picture with a weed range frame. The present invention can solve the problems of too low pulse emission frequency after conventional conversion, and difficulty in representing negative values ​​and biases. At the same time, the real-time and low power consumption of the model can be applied to embedded devices, mobile devices and unmanned machines with neuromorphic chips.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a low-power target detection method based on a convolutional pulse neural network. Background Art

[0002] Spiking neural networks are the third generation of neural networks. They use spiking neurons as computing units to simulate the encoding and processing of information in the human brain. Unlike traditional neural networks, spiking neural networks transmit information through a series of discrete peaks that form a spike sequence, rather than a continuous true value, thus providing sparse but powerful computing power. However, spiking neural networks have been limited to relatively simple image classification tasks and testing on smaller data sets, mainly due to the complex dynamics and non-differentiable operations of spiking neurons, and the lack of scalable training algorithms. In target detection, it is a great challenge to identify multiple targets and draw bounding boxes around them, and predicting the output value of the network requires high numerical precision. After the traditional deep neural network is converted to a spiking neural network, the performance is severely degraded when using a deep convolutional neural network for detection. The normalization method is too inefficient, resulting in a low pulse emission frequency. At the same time, in the field of spiking neural networks, there is no efficient leaky-Relu implementation, and it is ultimately difficult to detect any object. In the process of converting the original convolutional neural network to a spiking neural network, a certain layer of neurons in the convolutional neural network may output negative values, which are difficult to represent for spiking neural networks, and the bias of the convolutional neural network is also difficult to represent; at the same time, two layers of spiking neural networks are required to convert the Max-pooling layer in the convolutional neural network into the spiking neural network.

[0003] Weeds in farmland will compete with crops for nutrition, causing crop diseases and introducing pests. If not handled in time, it will cause serious agricultural losses. With the development of science and technology, today's weeding methods have gradually changed from manual weeding to machine weeding, but conventional machine weeding equipment has the problems of slow weed detection speed, low accuracy and high computing cost. Embedded and other machine equipment are difficult to perform weed detection and removal work on a large scale, for a long time and with high efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a low-power target detection method based on convolutional pulse neural network, which can enable machinery and equipment to detect weed targets around crops with high efficiency and low power consumption.

[0005] To achieve the above objectives, the present application proposes a low-power target detection method based on a convolutional spike neural network, comprising:

[0006] Establish a weed image dataset and perform preprocessing;

[0007] According to the deep convolutional neural network conversion conditions, a spiking neural network model for weed detection is constructed;

[0008] The pulse sequence of the spiking neural network model is used for neuron information transmission, and the accumulated input of the leaky integral firing model neurons is input into the membrane voltage;

[0009] Using the weed image dataset to train a spiking neural network model, achieve fine-grained normalization, and set model parameters;

[0010] The weed image to be identified is input into the trained spiking neural network model to obtain an image with a weed range frame.

[0011] Furthermore, a weed image dataset is established and preprocessed, specifically:

[0012] Weeds around different crop growth environments were photographed to create a weed image dataset. Based on the amount of data in the dataset, a generative adversarial network was used to obtain negative samples of the weed image dataset.

[0013] Image filtering, gamma correction and adaptive histogram equalization are used to preprocess weed images to reduce the interference of ambient light and noise on weed detection results.

[0014] Furthermore, according to the deep convolutional neural network conversion conditions, a pulse neural network model for weed detection is constructed, specifically:

[0015] Since there is a corresponding relationship between each pulse neural network unit and the deep convolutional neural network unit, under the condition of deep convolutional neural network, the input of neuron i at time t is Only when the membrane potential exceeds the threshold V thr ∈R + A pulse will be generated only when , and then the membrane potential is reset; therefore, for the neuron input of the lth layer, the pulse firing rate is obtained by establishing a connection between the input potential and the activation value of the deep convolutional neural network. as follows:

[0016]

[0017] Where i and j are the subscripts of the dimension where the parameter is located, M l-1 , are the number of neurons and neuron activation values ​​in the l-1th layer, are the weight and bias of the lth layer, r max is the maximum pulse emission rate, is the membrane potential of the ith neuron in layer l at time t, V thr is the threshold potential.

[0018] According to the leaky integral emission model based on the forward model, the spiking neural network model for weed detection is constructed as follows:

[0019]

[0020]

[0021] Among them, V represents membrane potential, o represents pulse output, t represents duration, and l represents the layer number; represents the delayed effect of membrane potential, W is the input synaptic weight, and f(·) is the step function.

[0022] Furthermore, the pulse sequence of the spiking neural network model is used for neuron information transmission, and the obtained leaky integral emission model neuron accumulation is input to the membrane voltage, specifically:

[0023] According to the proportional relationship between the pixel intensity and the pulse sequence in the weed image dataset, a fixed time interval is selected to generate a random number x[i] that conforms to the uniform distribution; at each time interval, if x[i]≤rΔt, a pulse is issued, otherwise no pulse is generated. The waiting time probability distribution p(τ) between two adjacent pulses is expressed as:

[0024]

[0025] Where τ is the time interval, r is the estimated value of the discharge frequency, d is the derivative function, and e -rτ It represents the probability that the number of pulses emitted is 0 within a time interval; when a pulse is emitted, a number is randomly selected from the number set that conforms to the exponential distribution as the waiting time for the next pulse emission; and so on, the pulse sequence of the pulse neural network model that conforms to the Poisson distribution is obtained;

[0026] The pulse sequence in the pulse neural network model is used to transmit neuronal information, and the obtained integral firing model neuron accumulates input to the membrane voltage V mem :

[0027]

[0028] represents the membrane voltage of the jth neuron in the lth layer, t is the time, is the weight of layer l at time t, when the membrane voltage V mem Greater than the threshold voltage V thr Generates pulse output

[0029] Furthermore, the spiking neural network model is trained using the weed image dataset to achieve fine-grained normalization and set model parameters, specifically: the resting potential and reset potential in the leaky integral emission model are set to 0, the integral time window and the kernel function representing the delay effect are set to 1, so that the gradient of the spiking neural network model is propagated along the time and space dimensions. In the process of training the spiking neural network using the weed dataset, the model parameter update is performed as follows:

[0030]

[0031]

[0032]

[0033] in is the derivative value from membrane potential V to output o, parameter a determines the gradient width, and L is the loss function.

[0034] Furthermore, in the spiking neural network model, let the connection relationship between the neuron and the lower layer be f out , when updating the neuron state for weed detection, only addition is performed, so that the power consumption is reduced. The total network operation amount within the simulation time T is:

[0035]

[0036] Among them, L is the number of neuron layers, S l (t) represents the number of pulses of neurons in layer l at time t.

[0037] Furthermore, in the process of training with the weed picture dataset, there are negative values ​​in the activation function layer that do not satisfy the pulse signal emission, so a signed neuron with an unbalanced threshold is set, and a critical voltage V is used in the positive region. th,pos , in the negative region, the threshold voltage V th,neg , passing positive and negative activation values, the formula is:

[0038]

[0039] Among them, α is a variable parameter, which is changed according to the specific activation function used.

[0040] Furthermore, the BN layer of the spiking neural network model introduces an affine transformation, so that the batch of weed image data optimized each time becomes 0 mean and 1 variance in the dimension of each layer channel, and the model weights are adjusted using the activation mean. and bias Normalize to:

[0041]

[0042]

[0043] Among them, u is the activation mean, is the standard deviation, and are the scale parameters and transformation parameters to be learned respectively.

[0044] Furthermore, the weed image to be identified is input into the trained spiking neural network model, the firing status of each neuron is corrected, the weeds in the input image are detected, and the use range box is marked.

[0045] Furthermore, in the process of weed detection by the spiking neural network model, the memory cost includes the weight memory M w 、Membrane potential memory p and pulse memory M s , the pulse memory M is calculated by the following formula:

[0046] M=M w +M p +M s

[0047] Among them, M w and M p Determined by the network model, and M s Dynamically determined by the maximum number of pulses per timestamp;

[0048] The consumption of the pulse convolution neural network model comes from the integration process of the pulse input. The integration time window and the kernel function representing the delay effect are set to 1, and the integration operation is changed to a pure addition operation. At the same time, the integration is calculated based on the pulse signal, so:

[0049] C=C add

[0050] Among them, C represents the total computational consumption of the model, C add Represents the integral operation consumption, which is proportional to the pulse event.

[0051] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: the weed detection model in the present invention is based on the conversion of a deep convolutional neural network into a pulse neural network, which can solve the problems of too low pulse emission frequency after conventional conversion, and difficulty in representing negative values ​​and biases. At the same time, the real-time and low power consumption of the model can be applied to embedded devices, mobile devices and unmanned machines with neuromorphic chips, so that the weeding machine can perform weeding work efficiently and for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1is a weed detection flow chart of the present invention;

[0053] Figure 2 It is a model transformation flow chart of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0055] Example 1

[0056] like Figure 1 As shown, the present application provides a low-power target detection method based on a convolutional pulse neural network, comprising:

[0057] S1: Establish a weed image dataset and perform preprocessing;

[0058] Specifically, we took photos of weeds around different crop growth environments to create a weed image dataset. Based on the amount of data in the dataset, we used a generative adversarial network to obtain negative samples of the weed image dataset, and obtained a dataset for model training and testing. We used image filtering, gamma correction, and adaptive histogram equalization to preprocess the weed images to reduce the interference of ambient light and noise on the weed detection results.

[0059] When performing histogram equalization, calculate the number of pixels n for each gray value in the weed image k And the grayscale cumulative distribution function S of the original image k , and obtain the grayscale transformation table; according to the grayscale transformation table, each grayscale level of the original image is mapped to a new grayscale level until the histogram equalization is completed.

[0060] S2: Based on the deep convolutional neural network conversion conditions, a spiking neural network model for weed detection is constructed;

[0061] Specifically, since there is a corresponding relationship between each pulse neural network unit and the deep convolutional neural network unit, under the condition of the deep convolutional neural network, the input of neuron i at time t is Only when the membrane potential exceeds the threshold V thr ∈R + A pulse will be generated when the membrane potential is reset; a threshold V is subtracted from the membrane potential. thr The reset mechanism, for the neuron input of the lth layer, establishes a connection between the input potential and the activation value of the deep convolutional neural network to obtain the pulse firing rate as follows:

[0062]

[0063] Where i and j are the subscripts of the dimension where the parameter is located, M l-1 , are the number of neurons and neuron activation values ​​in the l-1th layer, are the weight and bias of the lth layer, r max is the maximum pulse emission rate, is the membrane potential of the ith neuron in layer l at time t, V thr is the threshold potential;

[0064]

[0065] Where V thr is the threshold potential, for each neuron l, is a constant, and the membrane potential will increase by one at each subsequent time step. Improved efficiency in low activation situations.

[0066] When the membrane potential is higher than the threshold, the spiking neural network is regarded as a firing, at which point the output potential is 1, and the membrane potential returns to the reset potential; when the membrane potential is lower than the threshold, no firing occurs, and the output potential remains at 0; when the membrane potential is updated at each time step, the membrane potential drops by the difference between it and the resting potential, and rises by a contribution value transmitted by the neuron synapse. Therefore, according to the leaky integral emission model based on the forward model, the spiking neural network model for weed detection is constructed as follows:

[0067]

[0068]

[0069] Among them, V represents membrane potential, o represents pulse output, t represents duration, and l represents the layer number; represents the delayed effect of membrane potential, W is the input synaptic weight, and f(·) is the step function.

[0070] S3: The pulse sequence of the spiking neural network model is used for neuron information transmission, and the accumulated input of the leaky integral firing model neurons is input into the membrane voltage;

[0071] Specifically, according to the proportional relationship between the pixel intensity and the pulse sequence in the weed image dataset, a fixed time interval is selected to generate a random number x[i] in an interval that conforms to the uniform distribution; at each time interval, if x[i]≤rΔt, a pulse is issued, otherwise no pulse is generated. The probability distribution p(τ) of the waiting time between two adjacent pulses is expressed as:

[0072]

[0073] Where τ is the time interval, r is the estimated value of the discharge frequency, d is the derivative function, and e -rτ It represents the probability that the number of pulses emitted is 0 within a time interval; when a pulse is emitted, a number is randomly selected from the number set that conforms to the exponential distribution as the waiting time for the next pulse emission; and so on, the pulse sequence of the pulse neural network model that conforms to the Poisson distribution is obtained;

[0074] The pulse sequence in the pulse neural network model is used to transmit neuronal information, and the obtained integral firing model neuron accumulates input to the membrane voltage V mem :

[0075]

[0076] represents the membrane voltage of the jth neuron in the lth layer, t is the time, is the weight of layer l at time t, when the membrane voltage V mem Greater than the threshold voltage V thr Generates pulse output

[0077] S4: using the weed image dataset to train the spiking neural network, normalize the spiking neural network, and set model parameters;

[0078] Specifically, the resting potential and reset potential in the leaky integral firing model are set to 0, the integral time window and the kernel function representing the delay effect are set to 1, so that the gradient of the spiking neural network model is propagated along the time and space dimensions. In the process of training the spiking neural network using the weed dataset, the model parameter update is carried out as follows:

[0079]

[0080]

[0081]

[0082] in is the derivative value from membrane potential V to output o, parameter a determines the gradient width, and L is the loss function.

[0083] In the spiking neural network model, let the connection relationship between the neuron and the lower layer be f out , when updating the neuron state for weed detection, only addition is performed, so that the power consumption is reduced. The total network operation amount within the simulation time T is:

[0084]

[0085] Among them, L is the number of neuron layers, S l (t) represents the number of pulses of neurons in layer l at time t.

[0086] In the process of training with the weed picture dataset, the activation function layer has negative values ​​that do not satisfy the pulse signal emission. Therefore, a signed neuron with an unbalanced threshold is set, and the critical voltage V is used in the positive region. th,pos , in the negative region, the threshold voltage V th,neg , passing positive and negative activation values, the formula is:

[0087]

[0088] Among them, α is a variable parameter, which is changed according to the specific activation function used. Assume that V th,pos =1V, when α=0.1, V th,neg = -10V, the leaky-Relu can be efficiently and accurately imitated as follows:

[0089]

[0090] The BN layer of the spiking neural network model introduces an affine transformation, so that the batch of weed image data optimized each time becomes 0 mean and 1 variance in the dimension of each layer channel, and the model weights are adjusted using the activation mean. and bias Normalize to:

[0091]

[0092]

[0093] Among them, u is the activation mean, is the standard deviation, and are the scale parameters and transformation parameters to be learned, respectively. This makes it unnecessary to perform additional transformations on the BN layer after the BN layer is converted into the previous layer, and enables multiple neurons to emit pulses at a higher and more appropriate frequency.

[0094] S5: inputting the weed image to be identified into the trained spiking neural network model to obtain an image with a weed range frame;

[0095] Specifically, the weed image to be identified is input into the trained spiking neural network model, the firing condition of each neuron is corrected, the weeds in the input image are detected, and the use range box is marked.

[0096] The present invention can also evaluate the spiking neural network model based on the spiking memory and consumption, specifically:

[0097] Considering deploying the model on an embedded device, in the process of weed detection using the spiking neural network model, the memory cost includes the weight memory M w 、Membrane potential memory p and pulse memory M s , due to the ignition threshold u th and time constant τ can be shared by all neurons in the same layer and can therefore be ignored. When a spike is triggered, the spike memory M can be calculated by:

[0098] The pulse memory M is calculated by the following formula:

[0099] M=M w +M p +M s

[0100] Among them, M w and M p Determined by the network model, and M s Dynamically determined by the maximum number of pulses per timestamp;

[0101] In the process of weed target detection, the consumption of the pulse convolution neural network model comes from the integration process of the pulse input. By setting the integration time window and the kernel function representing the delay effect to 1, the complex multiplication calculation can be omitted, and the integration operation can be changed to a pure addition operation. At the same time, the integration is calculated based on the pulse signal, so:

[0102] C=C add

[0103] Among them, C represents the total computational consumption of the model, C add Represents the integral operation consumption, which is proportional to the pulse event.

[0104] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A low-power target detection method based on convolutional spike neural network, characterized in that: include: Establish a weed image dataset and perform preprocessing; According to the deep convolutional neural network conversion conditions, a spike neural network model for weed detection is constructed. Specifically, since there is a corresponding relationship between each spike neural network unit and the deep convolutional neural network unit, under the deep convolutional neural network conditions, the input of neuron i at time t is Only when the membrane potential exceeds the threshold V thr ∈R + A pulse will be generated only when , and then the membrane potential is reset; therefore, for the neuron input of the lth layer, the pulse firing rate is obtained by establishing a connection between the input potential and the activation value of the deep convolutional neural network. as follows: Where i and j are the subscripts of the dimension where the parameter is located, M l-1 , are the number of neurons and neuron activation values ​​in the l-1th layer, are the weight and bias of the lth layer, r max is the maximum pulse emission rate, is the membrane potential of the ith neuron in layer l at time t, V thr is the threshold potential; According to the leaky integral emission model based on the forward model, the spiking neural network model for weed detection is constructed as follows: Among them, V represents membrane potential, o represents pulse output, t represents duration, and l represents the layer number; represents the delayed effect of membrane potential, W is the input synaptic weight, and f(·) is the step function; The pulse sequence of the spiking neural network model is used for neuron information transmission, and the accumulated input of the leaky integral firing model neurons is input into the membrane voltage; Using the weed image dataset to train a spiking neural network model, achieve fine-grained normalization, and set model parameters; The weed image to be identified is input into the trained spiking neural network model to obtain an image with a weed range frame.

2. According to claim 1, a low-power target detection method based on convolutional spike neural network is characterized in that: Create a weed image dataset and perform preprocessing, specifically: Weeds around different crop growth environments were photographed to create a weed image dataset. Based on the amount of data in the dataset, a generative adversarial network was used to obtain negative samples of the weed image dataset. The weed images are preprocessed using image filtering, gamma correction and adaptive histogram equalization.

3. According to claim 1, a low-power target detection method based on convolutional pulse neural network is characterized in that: The pulse sequence of the spiking neural network model is used for neuron information transmission, and the accumulated input of the leaky integral firing model neuron is input to the membrane voltage, specifically: According to the proportional relationship between the pixel intensity and the pulse sequence in the weed image dataset, a fixed time interval is selected to generate a random number x[i] that conforms to the uniform distribution; at each time interval, if x[i]≤rΔt, a pulse is issued, otherwise no pulse is generated. The waiting time probability distribution p(τ) between two adjacent pulses is expressed as: Where τ is the time interval, r is the estimated value of the discharge frequency, d is the derivative function, and e -rτ It represents the probability that the number of pulses emitted is 0 within the time interval; when a pulse is emitted, a number is randomly selected from the number set that conforms to the exponential distribution as the waiting time for the next pulse emission; and so on, the pulse sequence of the pulse neural network model that conforms to the Poisson distribution is obtained; The pulse sequence in the pulse neural network model is used to transmit neuronal information, and the obtained integral firing model neuron accumulates input to the membrane voltage V mem : represents the membrane voltage of the jth neuron in the lth layer, t is the time, is the weight of layer l at time t. When the membrane voltage V mem Greater than the threshold voltage V thr Generate pulse output 4. According to claim 1, a low-power target detection method based on convolutional pulse neural network is characterized in that: The spiking neural network model is trained using the weed image dataset to achieve fine-grained normalization and set model parameters, specifically: the resting potential and reset potential in the leaky integral emission model are set to 0, the integral time window and the kernel function representing the delay effect are set to 1, so that the gradient of the spiking neural network model is propagated along the time and space dimensions. In the process of training the spiking neural network using the weed dataset, the model parameter update is performed as follows: in is the derivative value from membrane potential V to output o, parameter a determines the gradient width, and L is the loss function.

5. According to claim 4, a low-power target detection method based on convolutional pulse neural network is characterized in that: In the spiking neural network model, let the connection relationship between the neuron and the lower layer be f out , when updating the neuron state for weed detection, only addition is performed, where the total network operation amount within the simulation time T is: Among them, L is the number of neuron layers, S l (t) represents the number of pulses of neurons in layer l at time t.

6. According to claim 4, a low-power target detection method based on convolutional pulse neural network is characterized in that: In the process of training with the weed picture dataset, the activation function layer has negative values ​​that do not satisfy the pulse signal emission. Therefore, a signed neuron with an unbalanced threshold is set, and the critical voltage V is used in the positive region. th,pos , in the negative region, the threshold voltage V th,neg , passing positive and negative activation values, the formula is: Among them, α is a variable parameter, which is changed according to the specific activation function used.

7. The low-power target detection method based on convolutional spike neural network according to claim 1, characterized in that: The BN layer of the spiking neural network model introduces an affine transformation, so that the batch of weed image data optimized each time becomes 0 mean and 1 variance in the dimension of each layer channel, and the model weights are adjusted using the activation mean. and bias Normalize to: Among them, u is the activation mean, is the standard deviation, and are the scale parameters and transformation parameters to be learned respectively.

8. The low-power target detection method based on convolutional spike neural network according to claim 1, characterized in that: The weed image to be identified is input into the trained spiking neural network model, the firing of each neuron is corrected, the weeds in the input image are detected, and the use range box is marked.

9. A low-power target detection method based on convolutional spike neural network according to claim 8, characterized in that: In the process of weed detection by spiking neural network model, the memory cost includes weight memory M w 、Membrane potential memory p and pulse memory M s , the pulse memory M is calculated by the following formula: M=M w +M p +M s Among them, M w and M p Determined by the network model, and M s Dynamically determined by the maximum number of pulses per timestamp; The consumption of the pulse convolution neural network model comes from the integration process of the pulse input. The integration time window and the kernel function representing the delay effect are set to 1, and the integration operation is changed to a pure addition operation. At the same time, the integration is calculated based on the pulse signal, so: C=C add Among them, C represents the total computational consumption of the model, C add Represents the integral operation consumption, which is proportional to the pulse event.

Images