A low-delay local binary pattern neural network and a training method thereof
By introducing a precision loss estimator and binary weight block optimization into a binary spiking neural network, the problem of the binarization operation being unable to balance precision and spatial quantization is solved, achieving efficient network training and recognition capabilities while reducing computational costs.
Patent Information
- Application Number
- CN202310794000.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2043-06-30
AI Technical Summary
Existing binary spiking neural networks cannot achieve a balance between accuracy and spatial quantization during weight binarization operations, and require a large number of time steps to simulate neural dynamics and encoded information, resulting in increased computational costs.
Design a low-latency local binary spiking neural network, comprising an input layer, end-to-end spiking convolutional blocks, a global average pooling block, and an output layer. Utilize a precision loss estimator to automatically select binary and non-binarized network layers, and perform weight approximation optimization through binary weight blocks, while adjusting the weights using an alternative gradient function.
It achieves a balance between accuracy and spatial quantization, reduces storage space and time step size, improves the network's recognition ability and training efficiency, and approaches the performance of a full-precision network.
Smart Images
Figure CN116796809B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to binary spiking neural networks and their training methods, specifically to a low-latency local binary spiking neural network with an accuracy loss estimator and its training method. Background Technology
[0002] Spiking neural networks (SNNs) are brain-inspired models with enhanced spatiotemporal information processing capabilities and computational energy efficiency. However, as network depth increases, the storage problem caused by network weights gradually attracts attention. Inspired by artificial neural network quantization techniques, binary spiking neural networks (BNNs) are introduced to address this storage issue. Due to the lack of suitable learning algorithms, spiking neural networks are typically obtained through a conversion from artificial neural networks to spiking neural networks, and their accuracy is limited by the trained artificial neural network. For binary spiking neural networks, the weights of the first and last layers are usually binarized based on experimental experience to reduce accuracy degradation. This method is generally effective but fails to achieve a balance between accuracy and spatial quantization. Furthermore, spiking neural networks typically require sufficient time steps to simulate neural dynamics and encoded information, and convergence takes a long time, leading to increased computational costs. Summary of the Invention
[0003] The purpose of this invention is to address the shortcomings of existing binary spiking neural networks, which cannot achieve a balance between accuracy and spatial quantization when performing binarization operations on the weights of the first and last layers based on experimental experience. Furthermore, spiking neural networks typically require sufficient time steps to simulate neural dynamics and encoded information, and take a long time to converge, leading to increased computational costs. Therefore, this invention provides a low-latency local binary spiking neural network and its training method.
[0004] To address the shortcomings of the existing technology, the present invention provides the following technical solution:
[0005] A low-latency local binary spiking neural network is characterized by comprising an input layer with the input and output connected sequentially, N end-to-end spiking convolutional blocks, a global average pooling block, and an output layer; where N is a positive integer.
[0006] The input layer is used to receive raw images from external input and encode them into pulse signals;
[0007] Each of the pulsed convolutional blocks includes a precision loss estimator, a pulsed convolutional layer, a batch normalization layer, a neuron, and an average pooling layer, with the input and output connected in sequence.
[0008] The precision loss estimator is used to determine whether the network weights are binarized, and the input of the precision loss estimator is connected to the output of the input layer; the pulse convolutional layer is used to extract features; the batch normalization layer is used to normalize the input for each batch; and the average pooling layer is used for downsampling and feature compression.
[0009] The global average pooling block includes a convolutional layer and a global average pooling layer whose inputs and outputs are connected in sequence. The input of the convolutional layer is connected to the output of the average pooling layer, and the output of the global average pooling layer is connected to the input of the output layer.
[0010] The output layer is used to classify the input pulse signal and obtain the prediction result.
[0011] Furthermore, the accuracy loss estimator includes a binary weight calculation module, a loss calculation module, and a weight error measurement factor calculation module connected in sequence to the input and output. The binary weight calculation module is used to calculate binary weights based on the full-precision weights. The loss calculation module is used to calculate the binarization loss, spatial loss, and propagation loss based on the binary weights. The weight error measurement factor calculation module is used to calculate the weight error measurement factor based on the binarization loss, spatial loss, and propagation loss to determine whether the network weights are binarized.
[0012] Meanwhile, this invention provides a training method for the aforementioned low-latency local binary spiking neural network, characterized by the following steps:
[0013] Step 1: Preprocess the original images of the original dataset; simultaneously, establish the aforementioned low-latency local binary spiking neural network and set the hyperparameters.
[0014] Step 2: Input the original dataset processed in Step 1 into the input layer of the low-latency local binary spiking neural network established in Step 1 for encoding operations;
[0015] Step 3: Input the original dataset processed in Step 2 into the precision loss estimator to calculate the binary weights. Calculate the binarization loss, spatial loss, and propagation loss based on the binary weights to obtain the weight error measurement factor.
[0016] Step 4: Based on the weight error measurement factor in Step 3, select some network layers as non-binarized layers, use full-precision weight propagation, binarize the network weights of the remaining network layers, use binary weight propagation, and finally obtain the prediction result through the output layer.
[0017] Step 5: Measure the prediction results of Step 4 using a loss function;
[0018] Step 6: During backpropagation, use an alternative gradient function to approximate the gradient of the network weights; optimize and adjust the network weights based on their gradients.
[0019] Step 7: Determine whether the precision fluctuation of the network weights is less than or equal to 1%. If yes, proceed to step 8; otherwise, return to step 2.
[0020] Step 8: Calculate the weight loss L according to the chain rule, and adjust the binary weights according to the product L×LR of the weight loss L and the learning rate LR.
[0021] Step 9: Determine whether the network weights obtained in Step 8 meet the accuracy requirements. If they do, complete the training; otherwise, return to Step 2.
[0022] Furthermore, step 3 specifically includes:
[0023] Step 3.1: Initialize the full-precision weight W as the starting point, and use n binarized weight blocks to perform binarization approximation on the full-precision weight to obtain the binary weight BW, as shown in the following formula:
[0024]
[0025] Where n is a positive integer less than or equal to 5, n i For B i The adjustment factor, B i This represents the binary weights obtained from the i-th binary weight block;
[0026] B i =sign(W-mean(W)+(in)std(W))
[0027] Among them, Mean() is the average value, std() is the standard deviation value, and sign() is the sign function;
[0028] Step 3.2: Evaluate the binary weights BW from Step 3.1 using the original dataset processed in Step 2. Calculate the binarization loss, spatial loss, and propagation loss respectively to obtain the weight error measurement factor, as shown in the following formula:
[0029]
[0030] Where R(x) is the weight error measure, F(x) describes the impact of each layer of forward and backward propagation on the overall propagation process; x is the exponent of the layer, and ∈ means equal to The factor is sumL, which represents the total number of network layers, and K, which represents the number of categories in the original dataset.
[0031] A l The binarization loss of the l-th layer is given by the following formula:
[0032]
[0033] in, It is the i-th full-precision weight of the l-th layer. It is the i-th approximate weight of the l-th layer. This represents the number of output channels in the l-th layer; l = 1, ..., L;
[0034] M l This is the spatial loss of the l-th layer, and the formula is as follows:
[0035]
[0036] in, A is obtained when the number of output channels of the l-th layer is equal to 1. l , A is obtained when the number of output channels of the l-th layer equals the total number of weights. l .
[0037] Furthermore, in step 4, the partial network layer is the 2 layers with the largest R(x) in step 3.2.
[0038] Furthermore, step 8 specifically includes:
[0039] Calculate the weight loss L according to the chain rule, and obtain the product L×LR of the weight loss L and the learning rate LR; select the n closest to L×LR. i B i , and its corresponding B i Change from non-zero to zero or from zero to non-zero.
[0040] Furthermore, step 5 specifically includes:
[0041] The mean squared error (MSE) is used as the loss function, which is the mean of the sum of squares of the differences between the predicted value f(x) and the target value y. The formula is as follows:
[0042]
[0043] Furthermore, in step 6, the alternative gradient function h(u) is as follows:
[0044]
[0045] Where u is the membrane voltage, sign() is the sign function, and V th is the threshold, and 'a' is a parameter that determines the sharpness of the curve.
[0046] Further, in step 1, the preprocessing of the original images in the original dataset specifically involves:
[0047] Step a: Fill the original images in the original dataset into squares, where the squares are 4 pixels in size;
[0048] Step b: Randomly crop the square from step a to obtain the cropped image;
[0049] Step c: Randomly flip the cropped image from step b horizontally with a probability of 0.5 to obtain the flipped image;
[0050] Step d: Perform channel-by-channel normalization on the flipped image from step c;
[0051] x_normalized = (x0 - mean) / std
[0052] Where x0 is the original value of the flipped image, mean is the mean of the channel, std is the standard deviation of the channel, and x_normalized is the standardized value of the flipped image.
[0053] Furthermore, in step 1, the threshold V in the hyperparameters th The learning rate (LR) is initialized to 0.001, with a value of 0.5.
[0054] Compared with the prior art, the beneficial effects of the present invention are:
[0055] (1) The present invention provides a low-latency local binary spiking neural network, comprising an input layer, N end-to-end spiking convolutional blocks, a global average pooling block, and an output layer; the present invention sets up an accuracy loss estimator in the spiking convolutional blocks, which is used to automatically select the binarized and non-binarized network layers during network training by estimating the influence of different network layers on network accuracy, and takes into account three factors: the error caused by binarization, the size of the weight storage space, and the influence of the layers during network propagation, so as to achieve a balance between accuracy and spatial quantization.
[0056] (2) The present invention provides a training method for a low-latency local binary spiking neural network. First, n binary weight blocks are used to perform a binary approximation of the full-precision weights and classify it as a problem of finding the optimal weight coefficients. Second, when the neural network training tends to be stable, since coefficient optimization can no longer meet the requirements of improving accuracy, the present invention can further improve accuracy by adjusting the binary weights, thereby reducing the error caused by the binary weights, further reducing the network loss that has reached the training bottleneck, and making the network performance close to that of a full-precision network.
[0057] (3) The present invention provides a training method for a low-latency local binary spiking neural network, which balances the quantization and recognition capabilities of the network, enabling the network to maintain the recognition capability equivalent to that of a full-precision network, and reducing a large amount of storage space, thereby allowing the network to obtain better recognition accuracy with a smaller time step. Attached Figure Description
[0058] Figure 1 This is a schematic diagram of the structure of a low-latency local binary spiking neural network according to the present invention;
[0059] Figure 2 This is a schematic diagram of the structure of the pulse convolution block in an embodiment of the present invention;
[0060] Figure 3 This is a schematic diagram of the structure of the global average pooling block in an embodiment of the present invention;
[0061] Figure 4 This is a flowchart of a training method for a low-latency local binary spiking neural network according to the present invention. Detailed Implementation
[0062] The present invention will be further described below with reference to the accompanying drawings and exemplary embodiments.
[0063] Reference Figures 1 to 3 A low-latency local binary spiking neural network includes an input layer with the input and output connected in sequence, N end-to-end spiking convolutional blocks, a global average pooling block, and an output layer; N is a positive integer.
[0064] The input layer is used to receive the original image from the external input and encode it as a pulse signal.
[0065] Each pulsed convolutional block consists of a precision loss estimator, a pulsed convolutional layer, a batch normalization layer, a neuron, and an average pooling layer, with the input and output connected in sequence.
[0066] The accuracy loss estimator is used to determine whether the network weights are binarized. The input of the accuracy loss estimator is connected to the output of the input layer. The accuracy loss estimator includes a binary weight calculation module, a loss calculation module, and a weight error measurement factor calculation module, which are connected in sequence from the input to the output. The binary weight calculation module is used to calculate the binary weights based on the full-precision weights. The loss calculation module is used to calculate the binarization loss, spatial loss, and propagation loss based on the binary weights. The weight error measurement factor calculation module is used to calculate the weight error measurement factor based on the binarization loss, spatial loss, and propagation loss to determine whether the network weights are binarized.
[0067] The pulsed convolutional layer is used to extract features. The number of convolutional kernels in the pulsed convolutional layer is the same as the number of its output channels. Each convolutional kernel consists of a neuron. In this embodiment, LIF neurons are used. The batch normalization layer is used to normalize the input for each batch. The average pooling layer is used for downsampling and feature compression.
[0068] The global average pooling block consists of a convolutional layer and a global average pooling layer, with the input and output connected sequentially. The input of the convolutional layer is connected to the output of the average pooling layer, and the convolutional layer is a single layer. The output of the global average pooling layer is connected to the input of the output layer, and the global average pooling layer is a single layer. The convolutional layer contains neurons, and the number of neurons is the same as the number of convolutional kernels in the convolutional layer.
[0069] The output layer is used to classify the input pulse signal and obtain the prediction result.
[0070] Reference Figure 4 A training method for the aforementioned low-latency local binary spiking neural network includes the following steps:
[0071] Step 1: Preprocess the original images in the original dataset; simultaneously, based on LIF neurons, establish the low-latency local binary spiking neural network and set hyperparameters, including setting the threshold V. th The learning rate (LR) is initialized to 0.001, with a value of 0.5.
[0072] The preprocessing of the original dataset specifically involves:
[0073] Step a: Fill the original images in the original dataset into squares, with a size of 4 pixels.
[0074] Step b: Randomly crop the square from step a to obtain the cropped image;
[0075] Step c: Randomly flip the cropped image from step b horizontally with a probability of 0.5 to obtain the flipped image;
[0076] Step d: Perform channel-by-channel normalization on the flipped image from step c;
[0077] x_normalized = (x0 - mean) / std
[0078] Where x0 is the original value of the flipped image, mean is the mean of the channel, std is the standard deviation of the channel, and x_normalized is the standardized value of the flipped image.
[0079] Step 2: Input the original dataset processed in Step 1 into the input layer of the local binary spiking neural network established in Step 1 for encoding operations;
[0080] Step 3: Input the original dataset processed in Step 2 into the precision loss estimator to calculate the binary weights. Calculate the binarization loss, spatial loss, and propagation loss based on the binary weights to obtain the weight error measurement factor.
[0081] Step 3.1: Initialize the full-precision weight W as the starting point, and use n binarized weight blocks to perform binarization approximation on the full-precision weight to obtain the binary weight BW, as shown in the following formula:
[0082]
[0083] Where, n i For B i The adjustment factor, B i This represents the binary weights obtained from the i-th binary weight block;
[0084] B i =sign(W-mean(W)+(in)std(W))
[0085] Mean() calculates the average, std() calculates the standard deviation, and sign() is the sign function.
[0086] In this embodiment, n is taken as 3, and the binary weight BW can be written as:
[0087] BW≈n1B1+n2B2+n3B3
[0088] Where n1 = 0.3, B1 = 1, n2 = -0.5, B2 = 1, n3 = -1.1, B3 = 1;
[0089] Step 3.2: Evaluate the binary weights BW from Step 3.1 using the original dataset processed in Step 2. Calculate the binarization loss, spatial loss, and propagation loss respectively to obtain the weight error measurement factor, as shown in the following formula:
[0090]
[0091] Where R(x) is the weight error measure, F(x) describes the impact of each layer of forward and backward propagation on the overall propagation process; x is the exponent of the layer, and ∈ means equal to The factor is sumL, which represents the total number of network layers, and K, which represents the number of categories in the original dataset.
[0092] A l The binarization loss of the l-th layer is given by the following formula:
[0093]
[0094] in, It is the i-th full-precision weight of the l-th layer. It is the i-th approximate weight of the l-th layer. This represents the number of output channels in the l-th layer; l = 1, ..., L;
[0095] M l This is the spatial loss of the l-th layer, and the formula is as follows:
[0096]
[0097] in, A is obtained when the number of output channels of the l-th layer is equal to 1. l , A is obtained when the number of output channels of the l-th layer equals the total number of weights. l ;
[0098] Step 4: Based on the weight error measurement factor in Step 3, select the two layers with the largest R(x) in Step 3 as non-binarized layers, use full-precision weight propagation, binarize the network weights of the remaining network layers, use binary weight propagation, and finally obtain the prediction result through the output layer.
[0099] Step 5: Measure the prediction results of Step 4 using a loss function;
[0100] We choose to use the mean squared error as the loss function, which is the mean of the sum of squares of the differences between the predicted value f(x) and the target value y, as shown in the following formula:
[0101]
[0102] Step 6: During backpropagation, use an alternative gradient function to approximate the gradient of the network weights; adjust the network weights using stochastic gradient descent based on the gradient of the network weights.
[0103] The alternative gradient function is as follows:
[0104]
[0105] Where u is the membrane voltage, V th Here, is the threshold, and 'a' is a parameter that determines the sharpness of the curve; a = 1.
[0106] Step 7: Determine whether the precision fluctuation of the network weights is less than or equal to 1%. If yes, proceed to step 8; otherwise, return to step 2.
[0107] Step 8: Calculate the weight loss L according to the chain rule, and obtain the product L×LR of the weight loss L and the learning rate LR; select the n closest to L×LR. i B i , and its corresponding B iChange from non-zero to zero or from non-zero to non-zero, then proceed to step 9;
[0108] If the calculated value is L×LR=-0.45, then select n2B2 which is closest to L×LR and change B2 from 1 to 0;
[0109] Step 9: Determine whether the network weights obtained in Step 8 meet the accuracy requirements. If they do, complete the training; otherwise, return to Step 2.
[0110] In step 1, the LIF neuron information processing formula is as follows:
[0111]
[0112] in It is t i+1 The membrane voltage of the p-th neuron in the (l+1)-th layer. It is t i The output of the p-th neuron in the (l+1)-th layer is given by γ, where γ is the time constant and w is the output of the p-th neuron. pq l represents the weight of the q-th synapse on the p-th neuron. max It is the total number of neurons in the l-th layer.
[0113]
[0114] Among them, V th The threshold value is used.
Claims
1. A low-latency local binary pattern neural network, characterized by: The low-delay local binary spiking neural network comprises an input layer, N end-to-end pulse convolution blocks, a global average pooling block and an output layer, wherein N is a positive integer. The input layer is used for receiving an original image input from outside and encoding the original image into a pulse signal. Each pulse convolution block comprises a precision loss estimator, a pulse convolution layer, a batch normalization layer, a neuron and an average pooling layer connected in sequence. The precision loss estimator is used for determining whether the network weight is binarized, and the input end of the precision loss estimator is connected to the output end of the input layer. The pulse convolution layer is used for extracting features. The batch normalization layer is used for normalizing each batch of input. The average pooling layer is used for downsampling and feature compression. The global average pooling block comprises a convolution layer and a global average pooling layer connected in sequence. The output layer is used for classifying the input pulse signal to obtain a prediction result. The low-delay local binary spiking neural network is obtained by the following method: Step 1: preprocessing the original image of the original data set, and establishing the low-delay local binary spiking neural network and setting the hyperparameters at the same time; wherein n is a positive integer less than or equal to 5, n i is a scaling factor, B i is a scaling factor, B i represents a binary weight obtained by the i-th binary weight block; B i = sign(W - mean(W) + (i - n)std(W)) Step 2: inputting the original data set processed in step 1 into the input layer of the low-delay local binary spiking neural network established in step 1 for encoding operation; Step 3.1: inputting the original data set processed in step 2 into the precision loss estimator to calculate the binary weight: using n binary weight blocks to binarize and approximate the full-precision weight W to obtain the binary weight BW, and the formula is as follows: wherein Mean() is the average value, std() is the standard deviation, and sign() is the sign function; Step 3.2: calculating the binarization loss, spatial loss and propagation loss according to the binary weight, and then obtaining the weight error measurement factor; Step 4: selecting part of the network layer as a non-binarization layer according to the weight error measurement factor in step 3, using full-precision weight propagation, binarizing the network weight of the remaining network layer, using binary weight propagation, and finally obtaining the prediction result through the output layer; Step 8, calculate the weight loss L according to the chain rule, and obtain the product LxLR of the weight loss L and the learning rate LR; select the n closest to LxLR i B i , and its corresponding B i from non-zero to zero or from zero to non-zero; adjust the binary weight according to the product LxLR of the weight loss L and the learning rate LR, so as to reduce the error and network loss caused by the binary weight; Step 5: measuring the prediction result in step 4 by using the loss function; Step 6: in the process of back propagation, the gradient of the network weight is calculated by using the substitute gradient function; and the network weight is optimized and adjusted according to the gradient of the network weight; Step 7: determining whether the precision fluctuation of the network weight is less than or equal to 1%, if yes, executing step 8, otherwise returning to step 2 to continue to find the optimal weight coefficient; Step 9: determining whether the network weight obtained in step 8 meets the accuracy requirement, if yes, completing the training, otherwise returning to step 2.
2. The low-latency local binary pattern neural network of claim 1, wherein: The precision loss estimator comprises an output input sequentially connected binary weight calculation module, a loss calculation module and a weight error measurement factor calculation module, the binary weight calculation module is used for calculating binary weights according to full-precision weights, the loss calculation module is used for calculating binary loss, spatial loss and propagation loss according to binary weights, and the weight error measurement factor calculation module is used for calculating weight error measurement factors according to binary loss, spatial loss and propagation loss to determine whether the network weights are binarized.
3. The low-latency local binary pulse neural network of claim 1 or 2, wherein, The step 3.2 is specifically: The binary weight BW of step 3.1 is evaluated by using the original data set processed in step 2, and binary loss, spatial loss and propagation loss are calculated respectively, and then weight error measurement factors are obtained, and the formula is as follows: where R(x) is a weight error measurement factor, F(x) is used to describe the influence of each layer of forward propagation and back propagation on the overall propagation process; x is the index of the layer, ∈ is a factor equal to sumL represents the total number of network layers, and K represents the number of classes of the original data set; A l is the binarization loss of the first layer, and the formula is as follows: wherein, is the i-th full-precision weight of the l-th layer, is the i-th approximate weight of the l-th layer, is the number of output channels of the l-th layer; l = 1,..., L. M l is the spatial loss of the first layer, which is given by the following equation: wherein is A obtained when the number of output channels of the lth layer is equal to 1 l , is A obtained when the number of output channels of the lth layer is equal to the total number of weights l .
4. The low-latency local binary pattern neural network of claim 3, wherein: In step 4, the part of network layer is the 2 layers with the maximum R(x) in step 3.
2.
5. The low-latency, partial binary pulse neural network of claim 1, wherein, The step 5 is specifically: The mean square error MSE is used as the loss function, which is the mean value of the square sum of the difference between the predicted value f(x) and the target value y, and the formula is as follows:
6. The low-latency, partial binary pulse neural network of claim 1, wherein, In step 6, the substitute gradient function h(u) is as follows: where u is the membrane voltage, sign() is the sign function, V th is the threshold value, and a is a parameter that determines the sharpness of the curve.
7. The low-latency, partial binary pulse neural network of claim 1, wherein, In step 1, the pre-processing of the original image of the original data set is specifically: Step a, the original image in the original data set is filled into a square, and the size of the square is 4, unit: pixel; Step b, the square of step a is randomly cropped to obtain a cropped image; Step c, the cropped image of step b is randomly horizontally flipped with a probability of 0.5 to obtain a flipped image; Step d, the flipped image of step c is normalized channel by channel; x_normalized=(x0-mean) / std Where, x0 is the original value of the flipped image, mean is the mean value of the channel, std is the standard deviation of the channel, and x_normalized is the normalized value of the flipped image.
8. The low-latency, partial binary pulse neural network of claim 1, wherein: In step 1, the threshold V in the hyperparameters th The learning rate (LR) is initialized to 0.001, with a value of 0.5.