A storage optimization method for an impulse convolutional neural network accelerator
By constructing a pulse-less convolutional neural network equipped with time and space mask modules and optimizing storage access, the problem of high energy consumption of the pulse convolutional neural network accelerator is solved, and a significant reduction in energy consumption and computational complexity is achieved.
Patent Information
- Application Number
- CN202510340922.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-21
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2045-03-21
AI Technical Summary
Existing pulse convolutional neural network accelerators have high energy consumption and low energy efficiency while maintaining high accuracy. How to optimize storage to reduce energy consumption and computing power?
By constructing a pulse convolutional neural network with few pulses and adding time and space mask modules, storage access is optimized and the number of pulses and memory access times are reduced.
The energy consumption and computational complexity of SCNN were significantly reduced, with accuracy loss less than 1%, memory access energy consumption reduced by 20.04%-54.93%, and total energy consumption decreased by 50.48%.
Smart Images
Figure CN120146124B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of neural network processors, and particularly relates to a storage optimization method for a spiking convolutional neural network accelerator. BACKGROUND
[0002] No matter artificial neural networks or spiking neural networks, processing complex tasks often consumes more computing resources, and the number of neurons is increased and the network structure is deepened to improve network accuracy. In order to cope with the demand for efficient processing of complex tasks, in recent years, people have gradually deepened the research on spiking convolutional neural networks (SCNN). In addition, the field also generally designs SCNN accelerators as special computing chips, which have higher energy consumption and computing efficiency than general computing chips such as CPUs and GPUs. At present, the research target of SCNN mainly focuses on improving network accuracy, and the advantages of low energy consumption and high energy efficiency of spiking neural networks are often ignored in the process of improving accuracy. Especially for SCNN inference accelerators, how to make the SCNN accelerator still play the advantages of high energy efficiency while maintaining high accuracy is a challenge faced by SCNN research.
[0003] Similar to the time-coded shallow spiking neural network (Vanilla spiking neural network, VSNN), SCNN often uses the LIF neuron model. The difference is that SCNN has a network structure similar to convolutional neural networks, and its weight scale is more massive, and in the convolution operation of its accelerator, as long as there is a pulse in the input feature map of a channel, the corresponding weight value of this channel cannot be avoided. SUMMARY
[0004] The problem to be solved by the application is to reduce the access to heavy weights, and a storage optimization method for a spiking convolutional neural network accelerator is proposed.
[0005] To achieve the above purpose, the application realizes the following technical scheme:
[0006] A storage optimization method for a spiking convolutional neural network accelerator, comprising the following steps:
[0007] S1. Constructing a spiking convolutional neural network containing a few pulses of pulse rate, training the spiking convolutional neural network to obtain a few pulses of the spiking convolutional neural network accelerator;
[0008] S2. For the pulse-less convolutional neural network accelerator obtained in step S1, equip each neuron with a time mask module, where the time mask module consists of a comparator and an AND gate, and design a time masking method to perform time masking processing on the pulse-less convolutional neural network.
[0009] S3. For the pulse convolutional neural network accelerator with few pulses after time mask processing in step S2, a spatial mask module is equipped for each neuron. The spatial mask module consists of a counter and a comparator. A spatial mask method is designed to perform spatial mask processing on the pulse convolutional neural network accelerator with few pulses after time mask processing, and complete a storage optimization for pulse convolutional neural networks.
[0010] Furthermore, the specific implementation method of step S1 includes the following steps:
[0011] S1.1. On the basis of the original training objective function L, add a pulse rate multiplied by the preset coefficient d to obtain the objective function L containing the pulse rate s The expression is as follows:
[0012]
[0013] Among them, L is the original training objective function, which is the cross entropy between the output of the neural network and the training set label. is the average pulse rate, which is calculated using the number of pulses and time windows counted in the previous layer, and d is the preset coefficient used to adjust the pulse rate in L s the proportion in
[0014] S1.2. Using the objective function L containing the pulse rate s , the gradient descent method will be used for training, and the objective function will be calculated relative to the weight w and the neuron threshold voltage v th The partial derivative of , and the parameters are updated according to the value of the derivative. The calculation formula is as follows;
[0015] The calculation formula for weight W training is as follows:
[0016]
[0017] Among them, w ij is the element in the i-th row and j-th column of the weight matrix;
[0018] With w ij There is no functional relationship.
[0019] Neuron threshold voltage v th The training calculation formula is as follows:
[0020]
[0021] in, is the membrane potential of the i-th neuron in the l-th layer, t represents the time step, which means that the neuron is in a time window T w The time progress calculated within, 0≤t≤T w ; l represents the number of neuron layers, is the output pulse of the i-th neuron in the l-th layer, o represents the output pulse, for the neuron model adopted, a is the coefficient of the pulse emission process, which is set to 0.5. sign(·) represents the sign function, which outputs the sign of the input value of the sign function. If a positive number is input, the output is 1, otherwise the output is -1.
[0022] Furthermore, the hardware level design of step S2 is to equip each neuron in the SCNN with a time mask module. The time mask module consists of a comparator and an AND gate. The comparator has two input ports. The first input port is the output of the time step counter, and the second input port is the input value T. ap The output of the comparator and the output of the neuron are connected to the AND gate, and the output of the AND gate is used as the final output of the neuron.
[0023] Furthermore, the specific implementation method of the time mask method in step S2 is to add a comparator to the output end of the neuron in the SCNN, and make a judgment based on the time step t of the neuron. If it does not reach the set threshold T ap The neuron is allowed to reach v when the membrane potential u reaches th When a pulse is emitted, if the time step t has reached T ap , the neuron is prohibited from emitting pulses; T ap During the inference process, the calculation is performed according to the following formula:
[0024]
[0025] Among them, T w Represents the set time window, which is the input parameter of SCNN; It can be calculated during the inference calculation process of SCNN based on the output pulses of neurons before layer l.
[0026] Furthermore, the hardware level design of step S3 is to add a counter to each channel of the convolution layer, count the input pulses of each channel in a cumulative manner, and then compare the output C of the counter with the threshold of each convolution layer. Common input to a comparator, if The comparator outputs 1, indicating that the channel continues the convolution calculation; otherwise, the comparator outputs 0, indicating that the calculation of the channel is skipped; the output of the comparator accesses the memory access controller and the convolution layer calculation component, and the start / stop of the two components is controlled through the output signal of the comparator, so that the control of whether the convolution layer is skipped is realized.
[0027] Further, the specific implementation method of the space mask method in step S3 is that the SCNN has a convolution structure, the input data of the convolution layer is a feature map, the feature map is the output pulse of the previous layer, the convolution layer performs convolution operation on the input feature map and the weight value in units of channels, and the threshold value of each convolution layer is set for each convolution layer The number of input pulses of each channel is counted, when the number of input pulses is lower than the threshold value of each convolution layer, the channel is skipped, and the corresponding weight value access and convolution calculation are ignored; the threshold value of each convolution layer is calculated as follows:
[0028]
[0029] Wherein, f h is the height of the feature map input by the convolution layer.
[0030] Advantages of the present application:
[0031] The storage optimization method of the pulse-oriented convolutional neural network accelerator disclosed by the present application reduces the number of pulses of the SCNN and then further reduces the memory access times of the SCNN to complete the memory optimization method for reducing the memory access times of the SCNN. Since the memory access overhead of the SCNN accelerator accounts for a high proportion in the total energy consumption, the present application can significantly reduce the energy consumption of the SCNN by reducing the memory access overhead of the SCNN. On the other hand, the pulse reduction method contained in the present application can also reduce the number of pulses, thereby reducing the amount of calculation. The present application optimizes the energy consumption of the SCNN accelerator from the aspects of memory and calculation, and compared with the same type of method (pruning, quantization, etc.), the present application can achieve more energy consumption optimization effect with lower precision loss. BRIEF DESCRIPTION OF DRAWINGS
[0032] Figure 1 The flowchart of the storage optimization method of the pulse-oriented convolutional neural network accelerator disclosed by the present application;
[0033] Figure 2 The schematic diagram of the memory optimization mechanism disclosed by the present application;
[0034] Figure 3 The schematic diagram of the SCNN accelerator with the memory optimization mechanism disclosed by the present application. DETAILED DESCRIPTION
[0035] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the specific embodiments described are only a part of the embodiments of the present application, but not all the specific embodiments. The components of the specific embodiments of the present application generally described and shown in the drawings can be arranged and designed in various different configurations, and the present application can also have other embodiments.
[0036] Therefore, the detailed description of the specific embodiments of the present application provided below in the drawings is not intended to limit the scope of the claimed present application, but only represents selected specific embodiments of the present application. Based on the specific embodiments of the present application, all other specific embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0037] In order to further understand the inventive content, characteristics and effects of the present application, the following specific embodiments are exemplified, and the drawings are used for reference Figure 1 - the drawings Figure 3 The detailed description is as follows:
[0038] Example 1:
[0039] A storage optimization method for an impulse convolutional neural network accelerator, comprising the following steps:
[0040] S1. Constructing a few-impulse training of impulse rate for an impulse convolutional neural network, training the impulse convolutional neural network to obtain a few-impulse impulse convolutional neural network accelerator;
[0041] Further, the training of the impulse convolutional neural network is a training method in which the impulse rate is fused into the training objective function in a linear relationship;
[0042] Further, the specific implementation method of step S1 comprises the following steps:
[0043] S1.1. On the basis of the original training objective function L, add an impulse rate multiplied by a preset coefficient d to obtain an objective function L containing the impulse rate s The expression is as follows:
[0044]
[0045] Wherein, L is the original training objective function, is the cross entropy of the output of the neural network and the training set label, is the average impulse rate, which is calculated using the number of pulses and the number of time windows counted by the previous layer, d is a preset coefficient for adjusting the proportion of the impulse rate in L s .
[0046] S1.2. Utilize the target function L with pulse rate s The method of gradient descent will be used for training, calculating the partial derivative of the target function with respect to the weight w, neuron threshold voltage v th , and updating the parameters according to the size of the derivative value, the calculation formula is as follows;
[0047] The calculation formula of the weight W training is as follows:
[0048]
[0049] Where w ij is the element of the weight matrix in the i-th row and the j-th column;
[0050] There is no functional relationship between w ij ,
[0051] The calculation formula of the neuron threshold voltage v th training is as follows:
[0052]
[0053] Where, is the membrane potential of the i-th neuron in the l-th layer, t represents the time step, represents the time evolution calculated by the neuron in a time window T w , 0≤t≤T w ; l represents the number of layers of neurons, is the output pulse of the i-th neuron in the l-th layer, o represents the output pulse, for the adopted neuron model, a is the coefficient of fitting pulse emission process, set to 0.5, sign(·) represents the sign function, the positive and negative sign of the input value of the output sign function, if the input is positive, then the output is 1, otherwise the output is -1.
[0054] Further, the SCNN contains a large number of pulse neurons, the neurons have variables such as membrane potential u, threshold voltage v th , etc. The membrane potential u is a variable that gradually accumulates and rises as the neuron receives input pulses, when it reaches the set threshold voltage v th , the neuron sends a pulse to the neuron of the next layer and sets the membrane potential u to 0. Here the threshold voltage v th , like the weight W, affects the accuracy of the SCNN, so it can be solved by training to find the optimal value instead of manually inputting a fixed value. Under such a neuron model, the neuron threshold voltage v thThe training calculation formula needs to calculate the gradient of the objective function with respect to the membrane potential first, and then bring it into the calculation of the gradient of the objective function with respect to the threshold voltage v th .
[0055] S2. The pulse convolutional neural network accelerator of the few-pulse obtained in step S1 is equipped with a time mask module for each neuron, the time mask module is composed of a comparator and an AND gate, a time mask method is designed, and the few-pulse pulse convolutional neural network is processed by time mask;
[0056] Further, the hardware level design of step S2 is to equip each neuron in the SCNN with a time mask module, the time mask module is composed of a comparator and an AND gate, the comparator has two input ports, the first input port is the output of the time step counter, and the second input port inputs the value T ap , the output of the comparator is connected to the AND gate, and the output of the AND gate is used as the final output of the neuron;
[0057] Further, the specific implementation method of the time mask method designed in step S2 is that for the neurons in the SCNN, a comparator is added at the output end of the neuron, and the time step t performed by the neuron is judged according to the comparator. If the set threshold T ap is not reached, the neuron is allowed to fire a pulse when the membrane potential u reaches v th , if the time step t has reached T ap , the neuron is prohibited from firing a pulse; T ap is calculated in the reasoning process according to the following formula:
[0058]
[0059] Wherein, T w represents the set time window, which is an input parameter of the SCNN; can be calculated according to the output pulse of the neuron before the l-th layer in the reasoning calculation process of the SCNN; therefore, appropriate T ap can be set for each layer of neurons in real time.
[0060] Further, the time mask is different from directly shortening the length of the time window of the SCNN. Directly shortening the length of the time window will reduce the size of the internal transmission tensor, which will have a great influence on the reasoning accuracy; while the time mask only reduces the pulse and the calculation amount by suspending the work of the neuron, and does not change the size of the neuron output tensor, which has less influence on the accuracy.
[0061] S3. The pulse convolutional neural network accelerator for the few pulses after time mask processing is equipped with a spatial mask module for each neuron, which is composed of a counter and a comparator. A spatial mask method is designed to process the pulse convolutional neural network accelerator for the few pulses after time mask processing, and a storage optimization method for the pulse convolutional neural network is completed.
[0062] Further, the hardware level design in step S3 adds a counter to each channel of the convolutional layer, counts the input pulses of each channel in an accumulative manner, and then compares the output C of the counter with the threshold value of each convolutional layer The comparator outputs 1, indicating that the channel continues to perform convolution calculation; otherwise, the comparator outputs 0, indicating that the calculation of the channel is skipped; the output of the comparator is connected to the memory access controller and the convolutional layer calculation component, and the start / stop of the two components is controlled through the output signal of the comparator, so as to realize the control of whether the convolutional layer is skipped. The comparator outputs 1, indicating that the channel continues to perform convolution calculation; otherwise, the comparator outputs 0, indicating that the calculation of the channel is skipped; the output of the comparator is connected to the memory access controller and the convolutional layer calculation component, and the start / stop of the two components is controlled through the output signal of the comparator, so as to realize the control of whether the convolutional layer is skipped.
[0063] Further, the specific implementation method of the spatial mask method designed in step S3 is that the SCNN has a convolutional structure, the input data of the convolutional layer is a feature map, the feature map is the output pulse of the previous layer, and the convolutional layer performs convolution operation on the input feature map and the weight value in units of channels. The threshold value of each convolutional layer is set as follows: The number of input pulses of each channel is counted, and when the number of input pulses is lower than the threshold value of each convolutional layer, the channel is skipped, and the corresponding weight access and convolution calculation are ignored. The threshold value of each convolutional layer is calculated as follows:
[0064]
[0065] wherein, f h is the height of the feature map input by the convolutional layer.
[0066] The storage optimization method for the pulse convolutional neural network accelerator described in the embodiment is used to design the SCNN accelerator in the 28nm process library, and the Cadence Genus 15.0 software is used to evaluate the energy consumption and time delay. In addition, the accuracy of the ResNet18, ResNet34 and 7B-wideNet models is tested on the software platform based on the open-source PyTorch framework. The method proposed in the embodiment can reduce the pulses of the SCNN by 49.86%, reduce the energy consumption of the memory by 20.04%-54.93%, and reduce the total energy consumption by as high as 50.48%, while the accuracy loss of all tests is less than 1%.
[0067] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.
[0068] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and components may be substituted with equivalents without departing from the scope of the present application. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of these combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions within the scope of the claims.
Claims
1. A storage optimization method for a pulse convolutional neural network accelerator, characterized in that: The steps include: S1. Constructing a spiking convolutional neural network including few-pulse training of the pulse rate, training the spiking convolutional neural network to obtain a few-pulse spiking convolutional neural network accelerator; S2. For the pulse-less convolutional neural network accelerator obtained in step S1, equip each neuron with a time mask module, where the time mask module consists of a comparator and an AND gate, and design a time masking method to perform time masking processing on the pulse-less convolutional neural network. The hardware level design of step S2 is to equip each neuron in the SCNN with a time mask module. The time mask module consists of a comparator and an AND gate. The comparator has two input ports. The first input port is the output of the time step counter, and the second input port is the input value T. ap , the output of the comparator and the output of the neuron are connected to the AND gate, and the output of the AND gate is used as the final output of the neuron; S3. For the pulse-less convolutional neural network accelerator that has undergone temporal masking processing in step S2, equip each neuron with a spatial masking module, where the spatial masking module consists of a counter and a comparator. Design a spatial masking method to perform spatial masking processing on the pulse-less convolutional neural network accelerator that has undergone temporal masking processing, thereby completing a storage optimization for the pulse-less convolutional neural network. The hardware level design of step S3 is to add a counter to each channel of the convolution layer, count the input pulses of each channel in a cumulative manner, and then compare the output C of the counter with the threshold of each convolution layer. Common input to a comparator, if The comparator outputs 1, indicating that the channel continues to perform convolution calculations; otherwise, the comparator outputs 0, indicating that the calculation of the channel is skipped; the output of the comparator is connected to the memory access controller and the convolution layer calculation component of the memory, and the start / stop of these two components is controlled by the output signal of the comparator to realize the control of whether the convolution layer is skipped.
2. The storage optimization method for a pulse convolutional neural network accelerator according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1.
1. On the basis of the original training objective function L, add a pulse rate multiplied by the preset coefficient d to obtain the objective function L containing the pulse rate s The expression is as follows: Among them, L is the original training objective function, which is the cross entropy between the output of the neural network and the training set label. is the average pulse rate, which is calculated using the number of pulses and time windows counted in the previous layer, and d is the preset coefficient used to adjust the pulse rate in L s The proportion of S1.
2. Using the objective function L containing the pulse rate s , the gradient descent method will be used for training, and the objective function will be calculated relative to the weight w and the neuron threshold voltage v th The partial derivative of , and the parameters are updated according to the value of the derivative. The calculation formula is as follows; The calculation formula for weight W training is as follows: Among them, w ij is the element in the i-th row and j-th column of the weight matrix; With w ij There is no functional relationship. Neuron threshold voltage v th The training calculation formula is as follows: in, is the membrane potential of the i-th neuron in the l-th layer, t represents the time step, which means that the neuron is in a time window T w The time progress calculated within, 0≤t≤T w ; l represents the number of neuron layers, is the output pulse of the i-th neuron in the l-th layer, o represents the output pulse, for the neuron model adopted, a is the coefficient of the pulse emission process, which is set to 0.
5. sign(·) represents the sign function, which outputs the sign of the input value of the sign function. If a positive number is input, the output is 1, otherwise the output is -1.
3. The storage optimization method for a pulse convolutional neural network accelerator according to claim 2, characterized in that: The specific implementation method of the time mask method in step S2 is to add a comparator to the output end of the neuron in the SCNN, and make a judgment based on the time step t of the neuron. If it does not reach the set threshold T ap The neuron is allowed to reach v when the membrane potential u reaches th When a pulse is emitted, if the time step t has reached T ap , the neuron is prohibited from emitting pulses; T ap During the inference process, the calculation is performed according to the following formula: Among them, T w Represents the set time window, which is the input parameter of SCNN; It can be calculated during the inference calculation process of SCNN based on the output pulses of neurons before layer l.
4. The storage optimization method for a pulse convolutional neural network accelerator according to claim 3, characterized in that: The specific implementation method of the spatial mask method designed in step S3 is that SCNN has a convolutional structure. The input data of the convolutional layer is the feature map, and the feature map is the output pulse of the previous layer. The convolutional layer convolves the input feature map with the weight in units of channels, and sets the threshold of each convolutional layer for each convolutional layer. The number of input pulses for each channel is counted. When the number of input pulses is lower than the threshold of each convolutional layer, the channel is skipped and the corresponding weight access and convolution calculation are ignored. The threshold of each convolutional layer is calculated as follows: Among them, f h is the height of the feature map input to the convolutional layer.