A compression method and device for regularized bit serial calculation of neural network

Abstract

The present invention discloses a compression method for regularized bit-serial calculation of a neural network, comprising: step 1, quantizing weights and activation values ​​in a trained full-precision neural network model; step 2, performing guided bit-sparse training on the quantized neural network model; step 3, performing group structured constraint training on the bit-sparse trained neural network model to solve the problem of uneven distribution of non-zero bits in weights; step 4, deploying the trained neural network model on a neural network accelerator with a structured bit-serial multiplication component. The method solves the problem that although the bit-serial accelerator in the prior art can improve the bit sparsity of weights, the distribution of bit sparsity in the weights is still uneven, and there is also an unbalanced load on the computing units in the accelerator.

Description

Technical Field

[0001] The present invention relates to the field of embedded data intelligent processing and artificial intelligence technology; in particular, to a compression method and device for regularized bit serial calculation of a neural network. Background Art

[0002] Neural networks have become powerful algorithms in fields like artificial intelligence and computer vision, and their applications are expanding. However, neural networks require enormous computational resources and storage space, making them challenging to implement on various edge or embedded devices with limited performance and power consumption. Consequently, various neural network accelerator technologies have emerged.

[0003] The bit-serial neural network accelerator replaces the parallel multipliers in the traditional neural network accelerator with AND gates, shifters and adders. Each cycle performs a 1-bit multiplication and accumulation operation. See "Accelerator for Deep Neural Networks CN201780039356.8". The number of operating cycles of this method is proportional to the number of quantized bits of neurons in the neural network. Therefore, the acceleration effect can be achieved by reducing the number of quantized bits.

[0004] Since the bits that are 0 in the multiplier have no effect on the multiplication result, they can be skipped. "A neural network model compression method and device CN202010306190.5" achieves the purpose of skipping more invalid operations by compressing the neural network model to increase the bit sparsity of the weights. The existing bit-serial accelerator has a barrel effect, that is, when multiple weights are calculated in parallel in the accelerator, the accelerator forces the first completed operation unit to wait until all operation units have completed the current round of multiplication and accumulation operations in order to synchronize the various operation units, and then jump to the next round of operations together. Therefore, the running time of this round is determined by the weight with the longest time required, that is, the weight with the lowest bit sparsity. Although "A neural network model compression method and device CN202010306190.5" can improve the bit sparsity of weights, the distribution of bit sparsity in the weights is still uneven, and there is also a problem of unbalanced load of operation units in the accelerator. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a compression method and device for regularized bit-serial calculation of a neural network, so as to solve the problem that although the existing bit-serial accelerator can improve the bit sparsity of weights, the distribution of bit sparsity in the weights is still uneven, and there is also an unbalanced load problem of the computing unit in the accelerator.

[0006] Technical solution of the present invention:

[0007] A compression method for regularized bit-serial computation of a neural network, comprising:

[0008] Step 1: Quantize the weights and activation values ​​in the trained full-precision neural network model;

[0009] Step 2: Perform guided bit sparse training on the quantized neural network model;

[0010] Step 3: Perform group structured constraint training on the sparsely trained neural network model to solve the problem of uneven distribution of non-zero bits in weights;

[0011] Step 4: Deploy the trained neural network model to a neural network accelerator with a structured bit-serial multiplication unit.

[0012] The method of guided bit sparse training described in step 2 is: making guided modifications to the gradient after backpropagation in the traditional neural network gradient descent training algorithm; determining whether the update direction of the weight is increasing or decreasing by judging the positive or negative of the gradient, and then predicting the change in the number of non-zero bits that may be caused by the update by comparing the number of non-zero bits of the current weight and the number of non-zero bits of the adjacent weight in the update direction; if the direction of the update is towards a decrease in the number of non-zero bits, the intensity of the update is increased by increasing the gradient; conversely, if the direction of the update is towards an increase in the number of non-zero bits, the intensity of the update is reduced by decreasing the gradient.

[0013] The group structured constraint described in step 3 means: dividing the weights of the same input channel in the neural network into groups, and limiting the bit sparsity within each group by a threshold; if the number of non-zero bits in the binary weights within the group is more than the set threshold, bit pruning is performed from the lowest bit to the highest bit until the threshold is met.

[0014] The group structured constraint is executed during each forward propagation, and the gradient of the group structured constraint operation is approximated by 1 during each backpropagation until the neural network converges.

[0015] The method for quantizing the weights and activation values ​​in step 1 is: uniform quantization with a constant quantization step size.

[0016] A compression device for regularized bit-serial calculation of a neural network, wherein a group-structured bit-serial multiplication component completes a group-threshold shift-accumulation operation within a group-threshold cycle. First, a group of weights is encoded into a group-threshold power exponent, i.e., non-zero bits in the binary representation are eliminated.

[0017] The network weights are represented using a signed binary code.

[0018] Signed binary encoding outputs a signed power exponent in each cycle to indicate the number of shift bits, including:

[0019] E. Start judging the binary number X from the lowest bit and find the position of the bit with the lowest bit being 1, which is recorded as n;

[0020] F. Determine whether the n+1th bit X[n+1] is 0;

[0021] G. If it is 0, output the power index n, and perform the n-th power operation on X minus 2; if it is 1, output the power index -n, and perform the n-th power operation on X plus 2;

[0022] H. Repeat the above process until X is all zero, completing the signed power encoding.

[0023] It also includes: an activation value selection signal is used to identify which weight in the original group the power exponent comes from, which is subsequently used to select the activation value; secondly, a shift operation is performed on the selected activation value, which replaces the original multiplication operation; finally, the result is accumulated and the final output is calculated after the group threshold shift and accumulation operations.

[0024] More than one multiplier-accumulator is used to improve computing power. The multipliers-accumulators perform calculations in parallel and output results alternately. The number of multipliers-accumulators is determined by the hardware area and the group threshold required by the network.

[0025] Beneficial effects of the present invention:

[0026] This invention quantizes the weights and activations in a pre-trained full-precision neural network model, performs guided bit-sparse training, determines the sparsity of each layer in the neural network, and then applies group-structured constraints to the bit sparsity to create a group-structured bit-sparse model. Furthermore, the neural network can be deployed on the hardware accelerator proposed in this invention to improve energy efficiency.

[0027] The structured bit-serial multiplication unit structure of the present invention improves the computing capability and can effectively reduce power consumption.

[0028] The problem that the existing bit serial accelerator can improve the bit sparsity of weights, but the distribution of bit sparsity in the weights is still uneven, and there is also an unbalanced load on the computing units in the accelerator is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the process of the present invention;

[0030] Figure 2 Schematic diagram of the structured bit serial multiplication unit of the present invention;

[0031] Figure 3 A schematic diagram of improving the computing power of the multiplication component according to the present invention; DETAILED DESCRIPTION

[0032] Existing serial accelerators can reduce operations by skipping operations with zeros in the bit stream. Therefore, improving the bit sparsity in the neural network model, that is, the proportion of zeros in the binary weights, can improve the network's computing efficiency. However, simply improving the network's bit sparsity is not enough, as there is also the problem of uneven sparsity distribution. For example, when multiple groups of data are operated on simultaneously on a serial operation unit, the operation time of each group of data is different due to the different number of 1 bits contained in each group. In order to ensure synchronization between the various groups of operation units, the accelerator will force the group that completes first to wait for the group that completes later until all operations are completed before jumping to the next batch of data. Such a mechanism will lead to a great waste of resources.

[0033] The method of the present invention ensures that the running time of each group is the same, and also improves the network sparsity.

[0034] Figure 1 The method of the present invention comprises the following steps:

[0035] Quantize the weights and activations in pre-trained neural network models.

[0036] As an example, the pre-trained neural network model can be an ImageNet classification model, which is relatively common among deep neural network models. Specifically, it can be a ResNet18 classification model, a ResNet50 classification model, a VGG16 classification model, or a MobileNetv2 classification model. MobileNetv2 is a lightweight classification model with less redundant computation.

[0037] Current neural network quantization algorithms can ensure that neural network classification accuracy is similar to 32-bit floating-point results when using 16-bit or 8-bit fixed-point. Furthermore, in accelerator hardware implementation, using multiply-accumulators with lower quantization bits can save area and power. Therefore, the neural network is first quantized to 8 bits. This example uses uniform quantization with a constant quantization step size.

[0038] Perform guided bit-sparse training on quantized neural network models. Since bit-sparse computations are non-differentiable, they cannot be used as a component of the loss function during gradient descent training. This method guides the network toward bit-sparse updating by modifying the gradients after backpropagation in traditional neural network gradient descent training algorithms.

[0039] Specifically, by judging whether the gradient is positive or negative, it can be determined whether the update direction of the weight is increasing or decreasing, and then by comparing the number of non-zero bits of the current weight and the number of non-zero bits of the adjacent weight in the update direction, it is possible to predict the change in the number of non-zero bits that may be caused by the update. If the direction of the update is towards a decrease in the number of non-zero bits, the intensity of the update is increased by increasing the gradient. Conversely, if the direction of the update is towards an increase in the number of non-zero bits, the intensity of the update is weakened by reducing the gradient. Increasing or decreasing the gradient here is achieved by multiplying the gradient by a scaling factor. The scaling factor is calculated as the difference in the number of non-zero bits before and after the update divided by a hyperparameter, plus 1. This hyperparameter is used to adjust the balance between bit sparsity and neural network classification accuracy.

[0040] The bit sparsity of the neural network weights will increase as the above training method proceeds.

[0041] Group-structured constraint training is performed on a neural network model trained for bit sparsity to address the uneven distribution of non-zero bits in weights. This group-structured constraint involves dividing the weights of the same input channel in the neural network into groups and applying a threshold to constrain the bit sparsity within each group. If the number of non-zero bits in the binary weights within a group exceeds a set threshold, bit pruning is performed from the lowest to the highest bit until the threshold is met.

[0042] The group threshold here can be determined by the bit sparsity of each layer of the neural network. That is, the present invention believes that when the group size is sufficient, the sparsity within each group is consistent with the bit sparsity of the current layer of the neural network.

[0043] The reason for grouping weights for the same input channel is that weights within the same group will be accumulated to the same output after multiplication by the activation value. In terms of computational overhead, this is equivalent to replacing the group size multiplication and accumulation operations with the group threshold shift and accumulation operations.

[0044] Because bit pruning operations in group-structured constraints introduce errors into network weights, resulting in reduced neural network accuracy, it is necessary to retrain the neural network weights to be aware of group-structured constraints to restore network accuracy. This paper uses the Straight Through Estimator (STE). This means that group-structured constraints are applied during each forward pass, and the gradient of the group-structured constraint operation is approximated by 1 during each backward pass.

[0045] Group structured training of neural networks until convergence.

[0046] Deploy neural network models onto neural network accelerators with structured bit-serial multiplication units.

[0047] See also Figure 2The present invention provides a hardware group structured bit serial multiplication unit that completes a group threshold number of shift-accumulation operations within a group threshold number of cycles.

[0048] First, a set of weights is encoded as a group threshold power exponent, that is, non-zero bits in the binary representation are removed.

[0049] In order to further improve the bit sparsity of the network, a signed binary code is used to represent the network weights. Compared with the traditional unsigned binary representation of 0,1, this signed code can also be represented by -1. Booth code is a common signed code and is widely used in multipliers. The signed encoding method used in the present invention is different from Booth code. Each cycle outputs a signed power exponent to indicate the number of shift bits. The details are as follows:

[0050] 1. For the binary number X, start from the lowest bit and find the position of the bit with the lowest bit being 1, which is recorded as n.

[0051] 2. Determine whether the n+1th bit X[n+1] is 0.

[0052] 3. If it is 0, the power exponent n is output, and the operation of subtracting 2 to the power of n is performed on X; if it is 1, the power exponent -n is output, and the operation of adding 2 to the power of n is performed on X.

[0053] Repeat the above process until X is all zero, and the signed power encoding is completed.

[0054] There is also a corresponding activation value selection signal used to identify which weight in the original group the power exponent comes from, which will be used to select the activation value later.

[0055] Next, the selected activation values ​​are shifted, replacing the original multiplication operation. Finally, they are accumulated with the partial sums. After group threshold shift-and-accumulate operations, the final output is calculated.

[0056] Since parallel computing has been replaced by serial computing, computing power has decreased and computing latency has increased, so it is necessary to improve computing power.

[0057] See also Figure 3 This invention improves computing power by instantiating multiple multiplication and accumulation units (MACCs). These units can perform calculations in parallel and output results alternately. As long as the configured group threshold is less than or equal to the number of MACCs included, throughput can be achieved similar to parallel computing. In this case, the accelerator's bottleneck lies in memory access. The number of MACCs within the computational unit is primarily determined by the hardware area and the group threshold required by most networks. Equalizing the number of MACCs and group size can meet the group threshold requirements of most networks.

[0058] Here, the activation value is shared by all multipliers and accumulators. This method does not require modifying the memory architecture of traditional parallel multiplication, has good compatibility, and can effectively reduce power consumption by skipping invalid 0 operations.

Claims

1. A compression method for regularized bit-serial computation of a neural network, comprising: Step 1: Quantize the weights and activation values ​​in the trained full-precision neural network model; Step 2: Perform guided bit sparse training on the quantized neural network model; Step 3: Perform group structured constraint training on the sparsely trained neural network model to solve the problem of uneven distribution of non-zero bits in weights; Step 4: Deploy the trained neural network model to a neural network accelerator with a structured bit-serial multiplication unit; Among them, the method of guided bit sparse training described in step 2 is: making guided modifications to the gradient after backpropagation in the traditional neural network gradient descent training algorithm; determining whether the update direction of the weight is increasing or decreasing by judging the positive or negative of the gradient, and then predicting the change in the number of non-zero bits that may be caused by the update by comparing the number of non-zero bits of the current weight and the number of non-zero bits of the adjacent weight in the update direction; if the direction of the update is towards a decrease in the number of non-zero bits, the intensity of the update is enhanced by increasing the gradient; conversely, if the direction of the update is towards an increase in the number of non-zero bits, the intensity of the update is weakened by reducing the gradient; the increasing gradient or the decreasing gradient is achieved by multiplying the gradient by a proportional factor, the proportional factor is the difference between the number of non-zero bits before and after the update divided by a hyperparameter, plus 1, and the hyperparameter is used to adjust the balance between bit sparsity and neural network classification accuracy.

2. The compression method for regularized bit-serial computation of a neural network according to claim 1, characterized in that: The group structured constraint described in step 3 means: dividing the weights of the same input channel in the neural network into groups, and limiting the bit sparsity within each group by a threshold; if the number of non-zero bits in the binary weights within the group is more than the set threshold, bit pruning is performed from the lowest bit to the highest bit until the threshold is met.

3. The compression method for neural network regularized bit serial calculation according to claim 2, characterized in that: The group structured constraint is executed during each forward propagation, and the gradient of the group structured constraint operation is approximated by 1 during each backpropagation until the neural network converges.

4. The compression method for regularized bit-serial computation of a neural network according to claim 1, characterized in that: The method for quantizing the weights and activation values ​​in step 1 is: uniform quantization with a constant quantization step size.

5. A compression device for a neural network regularized bit-serial calculation corresponding to the compression method for a neural network regularized bit-serial calculation according to claim 1, characterized in that: The group-structured bit-serial multiplication unit completes group-threshold shift-accumulation operations within group-threshold cycles. First, a group of weights will be encoded as group-threshold power exponents, that is, non-zero bits in the binary representation are removed.

6. The compression device for neural network regularized bit serial calculation according to claim 5, characterized in that: The network weights are represented using a signed binary code.

7. The compression device for neural network regularized bit serial calculation according to claim 6, characterized in that: Signed binary encoding outputs a signed power exponent in each cycle to indicate the number of shift bits, including: A. Start judging the binary number X from the lowest bit and find the position of the bit with the lowest bit being 1, which is recorded as n. B. Determine whether the n+1th bit X[n+1] is 0; C. If it is 0, output the power index n, and perform the n-th power operation on X minus 2; if it is 1, output the power index -n, and perform the n-th power operation on X plus 2; D. Repeat the above process until X is all zero, completing the signed power encoding.

8. The compression device for neural network regularized bit serial calculation according to claim 7, characterized in that: It also includes: an activation value selection signal is used to identify which weight in the original group the power exponent comes from, which is subsequently used to select the activation value; secondly, a shift operation is performed on the selected activation value, which replaces the original multiplication operation; finally, the result is accumulated and the final output is calculated after the group threshold shift and accumulation operations.

9. The compression device for neural network regularized bit serial calculation according to claim 8, characterized in that: More than one multiplier-accumulator is used to improve computing power. The multipliers-accumulators perform calculations in parallel and output results alternately. The number of multipliers-accumulators is determined by the hardware area and the group threshold required by the network.

Images