Fine-grained per-vector scaling for neural network quantization

Abstract

Fine-grained per-vector scaling for neural network quantization is disclosed. Today, neural networks are used to enable self-driving cars and improve the quality of speech recognition, real-time language translation, and online search optimization. However, the operation of neural networks for the above applications consumes energy. Parameter quantization used by neural networks reduces the amount of memory required to store the parameters while also reducing power consumption during neural network operation. Matrix operations performed by neural networks require many multiplication computations, so reducing the number of bits multiplied can reduce the energy consumed. Quantizing smaller sets of parameters using shared scaling factors can improve accuracy compared to quantizing larger sets of parameters. By quantizing and scaling parameters using fine-grained per-vector scaling factors, the accuracy of the computations can be maintained. A vector includes one or more elements within a single dimension of a multi-dimensional matrix.

Claims

1. A computer implementation method for quantizing parameters of a neural network model, the method comprising: For each vector of parameters within a single dimension of a multidimensional parameter tensor, a per-vector scaling factor is computed to generate multiple per-vector scaling factors for the multidimensional parameter tensor, wherein the units of the vectors are set to match the units of the vector multiplication accumulation MAC circuit in a neural network accelerator. Calculate the coarse-grained scaling factor of the multidimensional parameter tensor, wherein the coarse-grained scaling factor is shared by at least two vectors of the parameters; For each vector of parameters, the precision of each parameter in the vector of parameters is reduced according to the corresponding scaling factor to generate a parameter vector with reduced precision. The reduced-precision parameter vector is processed by layers of the neural network model to generate output values; and The output value is modified by multiplying each output value by the coarse-grained scaling factor.

2. The computer implementation method according to claim 1, wherein the parameter is at least one of weight or activation.

3. The computer implementation method of claim 1, wherein the parameter is a weight, and the per-vector scaling factor is pre-calculated based on the weight.

4. The computer implementation method according to claim 1, wherein the parameter is calculated by one of post-training calibration or quantized perception retraining.

5. The computer-implemented method of claim 1, wherein, The parameter is an activation, and the per-vector scaling factor is dynamically calculated at each layer of the neural network model during inference.

6. The computer implementation method according to claim 1, wherein the process includes: Calculate the vector dot product for each vector of the parameters; The vector dot product is modified according to the per-vector scaling factor corresponding to each vector of the parameter to generate a scaled vector dot product of each vector of the parameter; as well as For each vector of the parameters, the scaled vector dot product is accumulated to generate elements of a multidimensional output tensor as the output value.

7. The computer-implemented method of claim 6, wherein the parameters comprise weights and the second multi-dimensional tensor of parameters comprises activations, and wherein, Modifying the vector dot product includes multiplying each vector dot product by a corresponding scaling factor for the weights used to calculate the vector dot product and a corresponding scaling factor for the activation.

8. The computer-implemented method of claim 6, wherein the parameters comprise weights and the second multi-dimensional tensor of parameters comprises activations, and wherein, Modifying the vector dot product includes: The corresponding scaling factor of the weights used to calculate the vector dot product is multiplied by the corresponding scaling factor of the activation to generate the scaling product factor. Rounding the scaling factor; and Each vector dot product is multiplied by a rounded scaling factor to generate the scaled vector dot product.

9. The computer implementation method according to claim 1, wherein the coarse-grained scaling factor is one of the scaling factors per layer, per tensor, or per channel.

10. The computer implementation method of claim 9, wherein the per-vector scaling factor is an integer number and the coarse-grained scaling factor is a floating-point number.

11. The computer implementation method of claim 1, wherein the steps of computation, reduction, and processing are performed in a server or data center to generate data, and the data is streamed to a user device.

12. The computer implementation method of claim 1, wherein the steps of computing, reducing, and processing are performed in a cloud computing environment.

13. The computer implementation method of claim 1, wherein the steps of calculating, reducing, and processing generate data, the data being used to train, test, or verify the neural network model or another neural network model used in a machine, robot, or autonomous vehicle.

14. The computer implementation method of claim 1, wherein the steps of calculation, reduction, and processing are performed on a virtual machine, the virtual machine including a graphics processing unit or a multi-core processor.

15. A system comprising: A processor configured to implement a neural network model, the neural network model being configured to quantize parameters in the following manner: - Calculate a per-vector scaling factor for each vector of the parameters within a single dimension of the multidimensional parameter tensor to generate multiple per-vector scaling factors for the multidimensional parameter tensor, wherein the units of the vectors are set to match the units of the vector multiplication accumulation MAC circuit in a neural network accelerator. - Calculate the coarse-grained scaling factor of the multidimensional parameter tensor, the coarse-grained scaling factor being shared by at least two vectors of the parameters; - For each vector of parameters, reduce the precision of each parameter in the vector of parameters according to the corresponding scaling factor to generate a reduced precision parameter vector; - The reduced-precision parameter vector is processed by layers of the neural network model to generate output values; and - Modify the output value by multiplying each output value by the coarse-grained scaling factor.

16. The system of claim 15, wherein the parameter is an activation and the per-vector scaling factor is dynamically computed at each layer of the neural network model during inference.

17. The system of claim 15, wherein the process comprises: Calculate the vector dot product for each vector of the parameters; The vector dot product is modified according to the per-vector scaling factor corresponding to each vector of the parameter to generate a scaled vector dot product of each vector of the parameter; as well as For each vector of the parameters, the scaled vector dot product is accumulated to generate elements of a multidimensional output tensor as the output value.

18. A non-transitory computer-readable medium storing computer instructions for quantizing parameters, which, when executed by one or more processors, cause the one or more processors to perform the following steps: For each vector of parameters within a single dimension of the multidimensional parameter tensor, a per-vector scaling factor is calculated to generate multiple per-vector scaling factors for the multidimensional parameter tensor, wherein, The unit of the vector is set to match the unit of the vector multiplication-accumulation MAC circuit in the neural network accelerator. Calculate the coarse-grained scaling factor of the multidimensional parameter tensor, wherein the coarse-grained scaling factor is shared by at least two vectors of the parameters; For each vector of parameters, the precision of each parameter in the vector of parameters is reduced according to the corresponding scaling factor to generate a parameter vector with reduced precision. The reduced-precision parameter vector is processed by layers of the neural network model to generate output values; as well as The output value is modified by multiplying each output value by the coarse-grained scaling factor.

19. The non-transitory computer-readable medium of claim 18, wherein the parameter is an activation and the per-vector scaling factor is dynamically computed at each layer of the neural network model during inference.

20. The non-transitory computer-readable medium of claim 18, wherein the process comprises: Calculate the vector dot product for each vector of the parameters; The vector dot product is modified according to the per-vector scaling factor corresponding to each vector of the parameter to generate a scaled vector dot product of each vector of the parameter; as well as For each vector of the parameters, the scaled vector dot product is accumulated to generate elements of a multidimensional output tensor as the output value.

Images