Quantization method for artificial neural network, system therefor, and artificial neural network device
The piecewise quantization method for artificial neural networks minimizes errors and enhances performance by dividing the quantization range into regions with tailored quantization functions, suitable for resource-constrained applications.
Patent Information
- Application Number
- JP2020156057
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-03-11
- Filing Date
- 2020-09-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-09-17
AI Technical Summary
Existing quantization techniques for artificial neural networks introduce quantization errors that reduce performance, particularly in resource-constrained applications, and require access to full training datasets for fine-tuning.
A piecewise quantization method that divides the quantization range into multiple regions, applying different quantization functions to each region to minimize errors, allowing for lower bit-width representations without full dataset access.
The method reduces quantization errors, improves performance, and lowers computational complexity, power consumption, and memory requirements, making it suitable for resource-constrained devices.
Smart Images

Figure 0007593761000023 
Figure 0007593761000024 
Figure 0007593761000025
Abstract
Description
[Technical field]
[0001] The present invention relates to quantization, and more particularly to a piecewise quantization method and system therefor and an artificial neural network device for artificial neural networks. [Background technology]
[0002] Artificial neural networks, such as deep neural networks (DNNs), are used for a variety of machine learning tasks, such as image classification and object detection. The accuracy of DNNs can be improved by extending one or more dimensions, such as network depth, network width, or image resolution, but this comes at the expense of increased computational complexity and memory requirements, as well as increased energy consumption and execution time. These increased resource requirements make it difficult to employ DNNs in resource-limited applications, such as embedded systems and mobile devices.
[0003] Quantization techniques can be used to reduce the resource requirements of DNNs by converting full-precision weights and activations to lower-precision (lower bit-width) representations. For example, while training an artificial neural network, commonly used 32-bit floating-point numbers (FP32) are quantized by converting them to smaller 8-bit integers (INT8), which reduces the number of discrete values used to represent each number. Such lower bit-width numbers take up less space in memory, which can reduce the complexity, cost, and / or energy consumption of the system (whether hardware, software, or a combination thereof) that processes them. However, quantization can introduce quantization errors that reduce the performance (accuracy) of the DNN.
[0004] To minimize quantization error and maintain the performance of the model at full accuracy, quantized DNNs are trained without prior knowledge using quantization-aware training. In other methods, pre-trained floating-point models are fine-tuned after quantization. However, both methods complicate the application process, slow it down, and require access to the full training dataset.
[0005] In some applications, post-training quantization (PTQ) using uniform quantization with INT8 numbers maintains nearly the performance of the original FP32 pre-trained model. However, in some applications, the resource requirements of 8-bit quantized DNNs may still be too high for widespread use. Lower bit-width techniques, such as 4-bit uniform PTQ, allow DNNs to be used for a wider range of resource-limited applications, but lower bit-width quantization may degrade performance to levels that cannot be accommodated. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] US Patent Application Publication No. 2017 / 0308789 [Patent Document 2] US Patent Application Publication No. 2019 / 0042935 [Patent Document 3] US Patent Application Publication No. 2017 / 0205736 [Patent Document 4] European Patent Application Publication No. 03543917 [Patent Document 5] International Publication No. 2019 / 006976 [Non-patent literature]
[0007] [Non-Patent Document 1] GUPTA, Suyog et al., “Deep Learning with Limited Numerical Precision”, arXiv:1502.02551 [cs.LG], 2015, pgs. 1-10 [Non-Patent Document 2] JACOB, Benoit et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2704-2713 [Non-Patent Document 3] JAIN, Shubham et al., “BiScaled-DNN Quantizing Long-tailed Datastructures with Two Scale Factors for Deep Neural Networks”, 2019 56th ACM / IEEE Design Automation Conference (DAC), 2-6 June 2019 [Non-Patent Document 4] KRISHNAMOORTHI, Raghuraman, “Quantizing deep convolutional networks for efficient inference: A whitepaper”, arXiv:1806.08342 [cs.LG], 21 Jun 2018, pg. 1-36 Summary of the Invention [Problem to be solved by the invention]
[0008] The present invention has been made in consideration of the above-mentioned problems in the conventional art, and an object of the present invention is to provide an artificial neural network quantization method and system, and an artificial neural network device, which minimizes quantization errors. [Means for solving the problem]
[0009] In order to achieve the above object, one aspect of the present invention provides a method for quantizing an artificial neural network, comprising the steps of dividing a quantization range for a tensor of the artificial neural network into a first region and a second region, and quantizing the values of the tensor in the first region separately from the values of the tensor in the second region.
[0010] The values of the tensor in the first domain may be quantized based on a first quantization function, and the values of the tensor in the second domain may be quantized based on a second quantization function. The first and second quantization functions may be different. The first and second quantization functions may be substantially identical. The first and second quantization functions may have different scale factors. The first and second quantization functions may have different offsets from each other. A linear quantization may be applied to the values of the tensor in the first domain and the second domain. The first region may include a positive portion and a negative portion. The second region may include a positive portion and a negative portion. The quantization range extends from the minimum to the maximum value of the tensor. Dividing a quantization range may include locating breakpoints for the first region and the second region. Locating the breakpoints may include determining a quantization error for at least a portion of the quantization range; and substantially minimizing the quantization error. Minimizing the quantization error may include formulating the quantization error as a function of positions of the breakpoints, formulating a first derivative of the position function, and determining values of the breakpoints at which the first derivative is substantially zero. The breakpoint values at which the first derivative is substantially zero may be determined using a binary search. The locations of the breakpoints can be approximated using regression. The quantization error can be substantially minimized using a grid search. The breakpoints may be located offline. The breakpoints may be located online. The breakpoints may be limited to be less than the midpoint of the quantization range. The breakpoints may be positioned to match the density of quantized values to the density of values of the tensor. The tensor may include at least two channels that are separately quantized in at least one of a first and a second domain. The tensor may include at least two groups that are separately quantized in at least one of a first and a second domain. The method may further include correcting a bias in the quantized values of the tensor. The tensor may include one or more weight values. The tensor may include one or more activations. The artificial neural network may include a deep neural network. The first region and the second region may be non-overlapping. The tensor may include learned values. Values of the tensor are generated in response to a training data set, and the quantization range of the tensor may be divided without accessing the full training data set or by accessing only a portion of the entire training data set.
[0011] In accordance with one aspect of the present invention, a system for achieving the above object includes a processor configured to execute a procedure, the procedure including dividing a quantization range for a tensor of an artificial neural network into a first domain and a second domain, and quantizing values of the tensor in the first domain separately from values of the tensor in the second domain.
[0012] The processor may be further configured to execute a procedure including locating breakpoints for the first region and the second region. The processor may be further configured to execute a procedure that includes locating the breakpoint by substantially minimizing a quantization error for at least a portion of the quantization range.
[0013] In order to achieve the above object, an artificial neural network apparatus according to one aspect of the present invention comprises a first accumulator configured to sum activations and weighted value products for a first region of a quantization range, a second accumulator configured to sum activations and weighted value products for a second region of the quantization range, and a third accumulator configured to sum activations for the second region.
[0014] The activation for the second region is offset. The first, second and third accumulators may be implemented in dedicated hardware. The first, second and third accumulators may be implemented with one or more processors configured to perform an accumulation procedure. Effect of the Invention
[0015] In accordance with the present invention, a method and system for piecewise quantization for neural networks that minimizes quantization error is provided. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a histogram diagram illustrating an example distribution of weight values in a trained deep neural network. [Diagram 2] 2 is a diagram showing the distribution of quantized values from 4-bit uniform post-learning quantization of the weight values shown in FIG. 1. FIG. [Diagram 3] FIG. 13 illustrates a distribution of quantized values from quantization after piecewise linear learning according to one embodiment of the present invention. [Figure 4] FIG. 13 illustrates another example of a piecewise quantization technique according to an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates an example of an apparatus for processing piece-wise quantized values in an artificial neural network according to an embodiment of the present invention. [Figure 6] FIG. 1 illustrates an example of a computing system in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific examples of the embodiments of the present invention will be described in detail with reference to the drawings.
[0018] The drawings are not necessarily drawn to scale, and elements of similar structure or function are generally designated by the same reference numerals throughout the drawings for example purposes. The drawings are presented to facilitate the description of the various embodiments described herein. The drawings do not describe every aspect of the embodiments described herein and do not limit the scope of the claims. To avoid obscuring the drawings, not all components, connections, etc. may be shown, and not all components have reference numerals. However, the pattern of component arrangement may be readily apparent from the drawings. The drawings, together with the specification, serve to illustrate embodiments of the invention and, together with the detailed description, to explain the principles of the invention.
[0019] In some artificial neural networks, the distribution of weight values and activations in the pre-trained model is approximated by Gaussian and / or Laplacian functions. Figure 1 shows an example of the distribution of weight values in a trained deep neural network (DNN). Most weight values are clustered around zero, while the remaining weight values are spread out in a long tail.
[0020] FIG. 2 illustrates the distribution of quantized values (illustrated by thick bars) from a 4-bit uniform post-training quantization (PTQ) of the weight values shown in FIG. 1. As is evident from FIG. 2, the uniform quantization technique divides the quantization range evenly despite the uneven distribution of weight values. Thus, especially in low bit-width regimes, the uniform quantization scheme assigns too few quantization levels to values with a large number of small sizes and assigns too many quantization levels to values with a relatively small number of large sizes. This can, for example, increase the quantization error, reducing the accuracy of the results and degrading the model's performance.
[0021] In an artificial neural network according to an embodiment of the present invention, the quantization range of a tensor, such as a weight value, is divided into two or more regions that are quantized separately. For example, a piecewise linear quantization technique divides the quantization range of a trained weight value distribution into two non-overlapping regions and applies uniform quantization to each region, as shown in FIG.
[0022] Figure 3 illustrates a distribution of quantized values (shown as thick bars) from piecewise linear post-learned quantization according to one embodiment of the present invention applied to the weight values shown in Figure 1. The height of each thick bar in Figure 3 represents the number of weight value samples quantized at that weight value.
[0023] The quantization range [-t, t] is divided into two regions, each having a positive part and a negative part. The first region includes the negative part [-x, 0] and the positive part [0, x]. The first region is represented as [-x, 0] ∪ [0, x] or alternatively as [-x, x]. The second region includes the negative part [-t, -x] and the positive part [x, t]. The second region is represented as [-t, -x] ∪ [x, t]. For convenience, the two parts of each of the two regions are also referred to collectively or individually as the four intervals of the quantization range [-t, -x], [-x, 0], [0, x], and [x, t].
[0024] The breakpoint x between the first and second regions is indicated by the dashed lines -x and x, and 2-bit uniform quantization is used within each interval, so there are 4 quantized values in each interval, for a total of 16 quantized values for the entire quantization range.
[0025] The thick bar at -x contains two coexisting bars, one at the top of the interval [-t, -x] and one at the bottom of the interval [-x, 0]. Similarly, the thick bars at 0 and x each contain two coexisting bars at the ends of adjacent intervals (piecewise) of the quantization range.
[0026] As is evident from Figure 3, the quantization values are more dense in the [-x, x] region (first region), where the distribution of weight values is even denser. Similarly, the quantization values are less dense in the [-t, -x] ∪ [x, t] region (second region), where the distribution of weight values is less dense. Depending on the implementation details, this density matching may allow the piecewise linear PTQ technique shown in Figure 3 to provide better performance (e.g., lower quantization error) than the uniform PTQ technique shown in Figure 2, even though both techniques (the piecewise linear PTQ technique and the uniform PTQ technique) use the same number of quantization values.
[0027] It should be noted that, depending on implementation details, piecewise quantization techniques according to embodiments of the present invention may provide one or more additional advantages, such as higher speed, lower power consumption, reduced memory requirements, lower hardware and / or software complexity and / or cost, reduced die area on an integrated circuit, implementation with little or no modification to commercial hardware, etc. For example, the uniform quantization scheme shown in Figure 2 requires 4-bit additions, multiplications, accumulations, etc., whereas the piecewise quantization technique shown in Figure 3 requires only 2-bit operations for at least a portion of the pipeline, which is faster and more efficient. Furthermore, depending on implementation details, some or all of these advantages may be realized without access to the full training dataset.
[0028] In one embodiment, the performance of the piecewise quantized model depends on the value of one or more breakpoints (e.g., in the embodiment of FIG. 3 ) being placed at or near one or more optimal locations to reduce or minimize quantization error. Embodiments of the invention also include techniques for locating one or more optimal breakpoints. These techniques include: formulating the quantization error as a function of the breakpoints, analytically solving for the zero derivative, approximating the optimal breakpoints with linear approximations, and / or performing a grid search, all of which are described in more detail below.
[0029] The embodiment shown in Figure 3 may be modified in a myriad of ways and adapted for use in numerous applications in accordance with embodiments of the present invention, some of which are described below.
[0030] In one embodiment, any bit-width is used to quantize any or all regions and / or portions of the regions of the quantization range, with different bit-widths being used for different regions and / or portions of the regions as well as different layers and / or nodes within the artificial neural network layers.
[0031] In one embodiment, piecewise quantization is applied to any tensor or combination of tensors in the artificial neural network, including weight values, activations (including before and after application of an activation function), and any other tensors or values to be quantized. Any tensor may be signed or unsigned. Piecewise quantization is applied to any tensor or combination of tensors with any distribution of values at any location in the artificial neural network. This includes tensors with any generally bell-shaped or normal distribution, such as a Gaussian or Laplacian distribution with a generally long tail. Piecewise quantization is applied to tensors with distributions of values that are uniform or non-uniform, symmetric or asymmetric, etc.
[0032] In one embodiment, piecewise quantization is applied to any type of artificial neural network, including deep neural networks, convolutional neural networks, feedforward neural networks, etc.
[0033] In one embodiment, piecewise quantization is used with any quantization function, including uniform or linear and non-linear quantization, affine or non-affine quantization, etc., or a combination thereof. Different quantization functions are used for different regions and / or portions of regions of the quantization range. Piecewise quantization is applied per-layer, per channel, per group, or any variation or combination thereof.
[0034] In one embodiment, the quantization range is divided into any number of regions and / or portions thereof, with any number of breakpoints between the regions and / or portions thereof. The regions and / or portions thereof are symmetric or asymmetric, balanced or unbalanced. The regions and / or portions thereof have only negative values, only positive values, or a combination thereof. The quantization ranges used for quantization cover the entire (minimum to maximum) range of values of the tensor, or cover only one or more portions of the entire range of values of the tensor. The quantization ranges are divided into regions and / or portions thereof that are overlapping or non-overlapping, adjacent or non-adjacent (touching or spaced apart), or any combination thereof. Depending on the implementation details, for example, the use of non-overlapping regions and / or portions thereof improves the accuracy of models using piecewise quantization according to embodiments of the present invention.
[0035] In one embodiment, quantization is applied separately to each region or portion of the region of the quantization range. In one embodiment, operations and / or devices are combined to fully or partially combine quantization in one or more regions or portions of the regions.
[0036] In one embodiment, piecewise quantization is performed online (in real time) or offline, or any combination thereof. Piecewise quantization is performed post-training, pre-training, tuning after initial training, or re-training, quantization-aware training, or any combination thereof. In one embodiment, piecewise quantization according to embodiments of the present invention is performed without access to the full training dataset that may have been used to generate the distribution of tensor values. In one embodiment, some or all of the training dataset is used. For example, a small portion of the training dataset is used to profile the range of activation functions.
[0037] In one embodiment, the piecewise quantization is implemented in hardware, software, or any combination thereof.
[0038] In one embodiment, piecewise quantization is implemented with affine quantization in one or more regions of the quantization range or portions of the regions. In one embodiment, affine quantization includes quantization parameters such as a scale factor and an offset or zero-point.
[0039] In one embodiment, the quantization is characterized by the following.
[0040]
number
[0041] where x q is the quantized output value, x is the input floating-point value, and x min is the floating-point value at the minimum end of the quantization region, and S is a scale factor given by:
[0042]
number
[0043] where x max is the floating-point value at the maximum end of the quantization region, and N levels =2 b where b is the bit width used for quantization.
[0044] In the case of symmetric quantization, x max is given by:
[0045]
number
[0046] In the case of asymmetric quantization, x max is given by:
[0047]
number
[0048]
number
[0049] where x is an array of input floating-point values.
[0050] Although embodiments of the present invention are not limited to the use of any particular type of quantization, the use of an affine quantization technique, such as the example described above with reference to mathematical Equation 1, may provide one or more advantages, depending on the implementation details. This is understood with reference to several alternative implementation techniques. For example, one possible technique for implementing quantization is to use a fixed-point representation of numbers. However, using fixed-point numbers imposes limitations on the smallest and / or largest numbers that can be represented, and on the options for selecting breakpoints and / or scale factors. Such limitations reduce the accuracy of the model and / or results. For example, the use of fixed-point numbers imposes limitations on the smallest and / or largest numbers that can be represented, and / or on the breakpoints and / or scale factors. N (where N is a positive or negative integer). In contrast, use of the quantization technique described above with reference to Equation 1 allows for the use of floating-point or other more flexible numerical representations for scale factors, offsets or zeros, breakpoints, etc., allowing for the representation of smaller numbers, wider ranges of numbers, etc. Depending on the implementation details, any of these factors may improve the accuracy of the model and / or results as well as other performance metrics.
[0051] A further potential advantage of the quantization technique, such as the example described above with reference to Equation 1, is that it facilitates the realization of non-overlapping quantization regions. For example, in one embodiment, the minimum end of the piecewise quantization region for larger values of x min By positioning x min There is no quantization resolution wasted on values smaller than , which are quantized as part of a separate region for smaller values.
[0052] Figure 4 illustrates another example of a piecewise quantization technique according to an embodiment of the present invention. The embodiment of Figure 4 shows how floating-point representations of weight values within the entire quantization range [-t, t] (indicated by the continuous horizontal lines) are mapped to four different 4-bit (INT4) quantized values in four intervals (piecewise) of the quantization range: [-t, -x], [-x, 0], [0, x], and [x, t].
[0053] In one embodiment, the optimal breakpoints are found by minimizing the expected quantization error. For example, in an embodiment with a Gaussian distribution of weight values and a quantization range [-t, t] divided into two regions (each region having a positive and negative part), the weight value distribution is divided into four intervals: [-t, -x], [-x, 0], [0, x], and [x, t]. Each interval has N levels (values) that indicate the quantized value of the floating point weight values. levels The Gaussian probability density function (PDF) is given by:
[0054]
number
[0055] And the cumulative distribution function (CDF) is given as follows:
[0056]
number
[0057] The expected quantization error is then given by:
[0058]
number
[0059] where C is a constant given by:
[0060]
number
[0061] where b is the bit width used for quantization.
[0062] The first derivative of Equation 7 is given as follows:
[0063]
number
[0064] Solving for 0 (zero) provides the optimal break point.
[0065] TIFF0007593761000011.tif15159 increases monotonically as follows:
[0066]
number
[0067]
number
[0068] Therefore, the unique optimal break point x opt exists as follows:
[0069]
number
[0070] This minimizes the quantization error.
[0071] Depending on the implementation details, the quantization error for interval uniform (linear) quantization techniques according to embodiments of the present invention may be expressed as: TIFF0007593761000015.tif9159 has a smaller quantization error than a uniform quantization process using the same number of quantization levels. For example, the piecewise quantization technique using 4-bit quantization for each of the four intervals of the quantization range according to an embodiment of the present invention has a smaller quantization error than a general quantization process that applies uniform 6-bit quantization to the same quantization range.
[0072] To locate the optimal breakpoint according to an embodiment of the present invention, the equation One approach to solving (or solving) TIFF0007593761000016.tif9159 is to solve it numerically, for example using a binary search with complexity O(log t). Depending on the implementation details, this approach can provide very accurate results and is particularly suitable for implementations where quantization is performed offline, for example when quantizing weight values.
[0073] Equations according to embodiments of the present invention Another approach to solving TIFF0007593761000017.tif10168 is to approximate the optimal break points using linear regression, which has O(1) complexity. For example, this can be accomplished by first determining the maximum value for the standard deviation of the weight values (or other tensor) to estimate the normalized quantization range [-t, t]. A binary search is then used to find optimal break points based on the normalized quantization range. The binary search results are then used to find a linear approximation to one or more optimal break points. In one embodiment, a Gaussian weight value distribution and small perturbations are used to approximate the optimal break points. Approximate optimal breakpoint value x for TIFF0007593761000018.tif10168 app is determined through such an approximation using the following formula:
[0074]
number
[0075] Depending on the implementation details, this approach, considered as a one-shot approach, can reduce the time required to determine the optimal breakpoints with little degradation of the quantization error. Therefore, the approximation approach is advantageous for implementations that use online or on-the-fly quantization, such as activation. This approach can be used with distributions other than Gaussian. For example, for a Laplacian distribution, x app The value of is determined as follows:
[0076]
number
[0077] Here, such a distribution is normalized.
[0078] Equations according to embodiments of the present invention Another approach to solving TIFF0007593761000021.tif10168 is to use a coarse-to-fine grid search, which does not require any assumptions about the distribution of tensor values. In one embodiment, a three-stage grid search uses the following stages to search for the optimal breakpoint ratio to the tensor's maximum value to minimize the quantization error: Step 1: Find the best ratio r1 with np.arange(0.1, 1.0, 0.1) Step 2: Find the best ratio r2 using np.arange(r1-0.05, r1+0.05, 0.01). Step 3: Find the best ratio r3 using np.arange(r2-0.005, r1+0.005, 0.001) Here, np is the NumPy operator.
[0079] Depending on the implementation details, the grid search approach provides very accurate results and is particularly suitable for implementations where quantization is performed offline. A potential advantage of the coarse-to-fine grid search approach is that it can directly reduce and / or minimize quantization errors without the need for specific knowledge and / or assumptions about the distribution of tensor values.
[0080] Table 1 shows some pseudocode for performing a coarse-to-fine grid search to find breakpoints according to an embodiment of the present invention.
[0081] [Table 1]
[0082] In one embodiment, when positioning breakpoints for piecewise quantization of a tensor having a generally bell-shaped distribution according to an embodiment of the present invention, the highest density of values generally clusters around 0 (zero). Therefore, it is advantageous to limit the breakpoint to less than the midpoint of the quantization range (e.g., x opt <t / 2), which is useful for finding optimal breakpoints using analytical techniques, search techniques, and / or other techniques that reduce or minimize quantization errors to place the breakpoints. In one embodiment, the quantization error is reduced or minimized by evaluating the quantization error for the entire quantization range. In other embodiments, the quantization error is reduced or minimized by evaluating the quantization error for a portion of the quantization range and / or for all or part of the intervals (piecewise) and / or regions into which the range is divided.
[0083] As discussed above, the quantization range may be divided into any number of regions and / or portions of the regions, with any number of breakpoints between the regions and / or portions of the regions. In embodiments with multiple breakpoints, any of the techniques described above may be extended to find one or more optimal breakpoint values. For example, in one embodiment, multiple optimal breakpoint values are determined by minimizing the expected quantization error given in Equation 7 above applied to the multiple breakpoints. As another example, the coarse-to-fine grid search may also be extended to locate multiple breakpoints. In one embodiment, determining the optimal number and / or location of breakpoints involves balancing a variety of considerations. For example, depending on implementation details, using multiple breakpoints may provide significant improvements in accuracy. However, in other implementations, the improvement in accuracy provided by multiple breakpoints may be less significant than the extra hardware and / or software execution time required to implement the additional breakpoints.
[0084] Although principles according to embodiments of the present invention do not require the use of any particular technique for finding one or more optimized breakpoints, any of the analysis, approximation, and / or search techniques described herein may provide a more easily analyzable, extensible, and / or robust solution.
[0085] In one embodiment, piecewise quantization is implemented in dedicated hardware according to an embodiment of the present invention. In a hardware implementation, piecewise quantization divides the range of a tensor into two or more regions, with each region implemented with a separate computation path, e.g., to accommodate different scaling factors and / or offsets. In an embodiment implementing piecewise linear quantization as shown in FIG. 3, the hardware implementation includes some hardware similar to that used for uniform quantization within each region and / or portion of that region (e.g., in intervals (piecewise) of the quantization range), but adds some equipment to accommodate the interval arrangement. For example, uniform quantization within each interval is implemented with two accumulators: a first accumulator performs the activation and tensor (e.g., weight value) product, and a second accumulator sums the activations. One additional accumulator is included to sum activations that are multiplied by a non-zero offset in intervals or regions that are offset from zero (eg, in region 2).
[0086] Thus, in one embodiment, a hardware implementation of the piecewise quantization technique according to an embodiment of the invention includes any or both of the following: adders and multipliers similar to those used for uniform quantization, three accumulation registers (first and second accumulation registers for summing the products in each of the two computation paths, and a third accumulation register for activation in the path that handles offsets other than zero), and one or two extra bits of storage per tensor (one representing the sign and one representing the range over which the tensor value was quantized). The sign bit is not included in implementations with unsigned values. The range bit is not included in multiply-accumulate computations (MACs) since it is only used when looking for the appropriate accumulator. Thus, in one embodiment, a hardware implementation of the piecewise quantization technique according to an embodiment of the invention is realized with minimal modifications to commercially available hardware.
[0087] Figure 5 illustrates an example of an apparatus for processing piecewise quantized values in an artificial neural network according to an embodiment of the present invention. In the embodiment of Figure 5, weight value buffer 112 includes a column of 4-bit quantized weight values, each accompanied by an index bit indicating whether it was quantized in region 1 (gray) or region 2 (white). Any number of columns may be used to provide parallel processing of multiple values, and the columns may have any suitable depth. In this example, the columns are 9 weight values deep. A local input feature map (IFM) queue 110 provides a stream of 4-bit quantized values (ifm_value), each accompanied by a most significant nibble / least significant nibble bit (ifm_MSN / LSN) indicating which region it is associated with. In the local IFM queue 110, an IFM index (ifm_idx) associated with each value (ifm_value) uses a selector 114 (in this example, a 1-of-9 selector) to select a weight value brick 120 from the local IFM queue 110 to which the value (ifm_value) is multiplied by a 4-bit multiplier 116. The partial product is shifted in parts of 4 or so bits in a shift register 118 depending on the state of the ifm_MSN / LSN bits. Thus, each weight value brick 120 is associated with a corresponding set 122 of IFM index (ifm_idx) and nibble bit (ifm_MSN / LSN) values. An offset adder tree 124 associated with each column sums activation if the weight value of the corresponding column belongs to region 2. In the example state shown in FIG. 5, the first column is associated with the first region. Thus, the corresponding offset adder tree 124 for the first column is deactivated as indicated by an x. In contrast, the second and last columns are associated with the second region, and therefore the corresponding offset adder trees 124 for the second and last columns are activated, as indicated by the circles.
[0088] The embodiments of the invention relating to piecewise quantization can be combined with other quantization techniques, thereby producing additional embodiments that produce synergistic results. For example, an embodiment implementing piecewise uniform (linear) quantization implements bias correction within each interval of the quantization range, either individually or collectively. In such an embodiment, in the quantized values of a tensor, bias is corrected, for example, by folding one or more correction terms into a scale and / or offset for one or more intervals of the quantization range. Depending on the implementation details, the combination of piecewise quantization bias correction can further reduce quantization error and improve the performance of the artificial neural network in which it is implemented.
[0089] As another example, an embodiment implementing piecewise uniform (linear) quantization according to an embodiment of the present invention applies individual quantization per output channel filter for any one or more of the intervals of the quantization range. This technique is further extended according to an embodiment of the present invention to decompose the filters of one or more output channels into one or more groups that are quantized separately for one or more intervals of the quantization range. Depending on the implementation details, a combination of per-channel and / or per-group piecewise quantization can further reduce quantization error and further improve the performance of the artificial neural network. In one embodiment, changing from one group to another includes changing the scale factor. Depending on the implementation details, applying piecewise quantization per channel (or per channel) can improve accuracy compared to a per-layer (or per-layer) implementation.
[0090] Any or all of the methods and / or apparatus according to embodiments of the present invention may be used in any application using artificial neural networks. By way of example, artificial neural networks implementing piecewise quantization according to embodiments of the present invention may be used in devices for mobile computing, Internet of Things (IOT), and / or embedded applications, including devices configured to use artificial neural networks implementing piecewise quantization for image classification, face recognition, speech recognition, semantic segmentation, autonomous driving, robotic control, and other applications. Artificial neural networks implementing piecewise quantization according to embodiments of the present invention may also be used in desktop / workstation applications, servers, data centers, including hyperscale data centers, and the like.
[0091] FIG. 6 illustrates an example of a computing system according to an embodiment of the present invention. The system 150 of FIG. 6 may be used to implement any or all of the methods and / or apparatus according to embodiments of the present invention. The system 150 includes a central processing unit 152 (CPU), a user interface 154, a network interface 156, a memory 158, a storage 160, a GPU 162, a field programmable gate array (FPGA) 157, and an application specific integrated circuit (ASIC) 163. However, in accordance with embodiments of the present invention, the implementation is not limited to any of the components illustrated in FIG. 6 and may be implemented in any suitable hardware, software, or combination thereof. In different embodiments, the system may omit any one of these components or include any other type of components for implementing any of the methods and / or apparatus according to embodiments of the present invention, as well as any duplication or number of any of the components.
[0092] The CPU 152 includes any number of cores, caches, buses and / or interconnect interfaces and / or controllers. The memory 158 includes any arrangement of DRAM and / or SRAM, non-volatile memory (e.g., flash memory), and the like. The storage 160 includes a hard disk drive (HDD), a solid state drive (SSD), and / or any other type or combination of data storage devices. The user interface 154 includes any type of human interface device, such as a keyboard, mouse, monitor, video capture or transmission device, microphone, speaker, touch screen, as well as any virtualized or remote version of such devices. The network interface 156 includes one or more adapters or other devices for communicating via Ethernet, Wi-Fi, Bluetooth, or any other computer networking configuration, allowing the components to communicate over physical and / or logical networks, such as an intranet, the Internet, a local area network (LAN), a wide area network (WAN), and the like. GPU 162, FPGA 157, and ASIC 163 include dedicated hardware, such as programmable multiply- accumulate computations (MAC) hardware, that can be configured to efficiently implement any or all of the quantization and / or other neural network processing according to embodiments of the present invention.
[0093] Any or all of the components of system 150 are interconnected via a system bus 164. System bus 164 refers to a variety of interfaces including a power bus, an address and data bus, high speed interconnects such as Serial AT Attachment (SATA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (PCI-e), System Management Bus (SMB), and other types of interfaces that allow components to operate locally from one location and / or distributed among different locations.
[0094] System 150 may also include various chipsets, interfaces, adapters, glue logic, programmable or non-programmable logic devices or arrays such as embedded controllers, ASICs, embedded computers, smart cards, etc., such that the various components of system 150 work together to implement any or all of the methods and / or apparatus according to embodiments of the present invention. Any component of system 150 may be implemented in hardware, software, firmware, or any combination thereof. In one embodiment, any or all components may be implemented in a virtualized manner and / or in a cloud-based implementation with elastic provisioning of resources, for example, within a data center or distributed across multiple data centers.
[0095] Portions or steps of methods or algorithms and functions according to embodiments of the invention may be embodied directly in one or more software modules executed by the hardware or processor comprising system 150, or a combination of the two. When implemented in software, the functions may be stored or transmitted on a tangible, non-transitory computer readable recording medium as one or more instructions or code. The software modules may be stored in a RAM (Random Access Memory), a flash memory, a ROM (Read Only Memory), an EPROM (Electrically Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a register, a hard disk, a removable disk, a CD-ROM, or other form of storage medium. Any system according to embodiments of the invention or components or portions thereof may be implemented as part of a software stack of a larger system. Any system according to embodiments of the invention or components or portions thereof may be implemented as a software stack itself.
[0096] While the above-described embodiments have been described in the context of various implementation details, embodiments of the invention are not limited to these or any other particular details. For example, while some functionality has been described as being implemented by particular components, in other embodiments, functionality is distributed among components that reside in different locations and have various user interfaces and different systems. Although particular embodiments have been described as having particular processes, steps, etc., such terms also include embodiments that are implemented in particular processes, steps, etc., or in multiple processes, multiple steps, etc., or integrated into a single process, single step, etc. Additionally, references to components or elements may refer to only a portion of the components or elements.
[0097] In this specification and claims, the use of terms such as "first" and "second" is intended to distinguish what they refer to only and may not imply any spatial or temporal order unless the context clearly indicates otherwise. Reference to a first thing may not imply the presence of a second thing.
[0098] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiment and can be modified in various ways without departing from the technical concept of the present invention. [Explanation of symbols]
[0099] 110 Local IFM (Input Feature Map) Queue 112 Weight Value Buffer 114 Selector 116 4-bit multiplier 118 Shift Register 120 Brick 122 Sets 124 Offset Adder Tree 150 Systems 152 Central Processing Unit 154 User Interface 156 Network Interface 157 FPGA 158 Memory 160 Storage 162 GPU 163 ASIC
Claims
1. A method for quantizing an artificial neural network executed by a processor, comprising: dividing a quantization range for a tensor of the artificial neural network into a first region and a second region; and linearly quantizing the values of the tensor in the first domain and the values of the tensor in the second domain separately, dividing the quantization range into a first region and a second region includes placing one or more breakpoints between the first region and the second region; the linear quantization step includes a step of dividing a quantization range of a weight distribution after learning into the first region and the second region that do not overlap, and then applying uniform quantization to each of the first region and the second region; the quantization range is divided into the first region and the second region, each region having a positive portion and a negative portion; A method according to claim 1, characterised in that within each interval within each of said regions of said quantisation range, a uniform quantisation of 2 bits is used.
2. The values of the tensor in the first region are quantized based on a first quantization function; 2. The method of claim 1, wherein the values of the tensor in the second domain are quantized based on a second quantization function.
3. 3. The method of claim 2, wherein the first quantization function and the second quantization function have different scale factors.
4. 4. The method of claim 3, wherein the first quantization function and the second quantization function have different offsets from each other.
5. The step of placing the breakpoints comprises: determining a quantization error for at least a portion of the quantization range; and minimizing the quantization error. The step of placing the breakpoints comprises: analytically solving for the zero derivative; approximating the optimal break points with a linear approximation method; performing a grid search.
6. The step of minimizing the quantization error comprises: formulating the quantization error as a function of the positions of the breakpoints; formulating a first derivative of the position function; and determining a breakpoint value at which the first derivative is zero.
7. 7. The method of claim 6, wherein the breakpoint values at which the first derivative is zero are determined using a binary search.
8. 6. The method of claim 5, wherein the placement of breakpoints is approximated using regression.
9. The method of claim 5 , wherein the quantization error is minimized using a grid search.
10. 2. The method of claim 1, wherein the breakpoints are positioned to match a density of quantized values and a density of values of the tensor.
11. The method of claim 1 , wherein the tensor includes one or more weight values.
Citation Information
Patent Citations
Dynamic adaptation of deep neural networks
EP3543917A1
Information processor, information processing method, and information processing program
JP2020177535A
Fixing member and method for producing same
US20170205736A1
Computing system for training neural networks
US20170308789A1
Dynamic quantization of neural networks
US20190042935A1
