Quantization range estimation for quantized training

Abstract

Certain aspects of the present disclosure provide techniques for efficient quantized learning. A tensor is received at a layer of a neural network, and a current tensor is generated at a first bit-width based on the received tensor. One or more quantization parameter values are determined based on the current tensor. The current tensor is quantized to a lower bit-width in accordance with the one or more quantization parameter values determined based on a previous tensor generated during training of the neural network.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Greek patent application No. 20210100273, filed on April 16, 2021, the entire contents of which are incorporated herein by reference.

[0003] introduction

[0004] This disclosure relates in various aspects to machine learning, and in particular to the quantized training of machine learning models.

[0005] Deep neural networks (DNNs) have been widely used in various applications, such as image recognition, object detection, and machine translation. However, to improve the accuracy and effectiveness of these networks, their size has increased dramatically. Processing such large models incurs high computational costs and memory usage, which hinders the deployment of such networks on resource-constrained devices, such as smartphones, wearable devices, or drones, to name a few.

[0006] In recent years, low-order network quantization for neural network inference has been studied to reduce the computational complexity on the inference side. However, training such models still mainly relies on full-precision floating-point formats.

[0007] Therefore, more efficient quantitative training techniques are needed.

[0008] Brief Overview

[0009] Some aspects provide a method including: generating a current tensor with a first bit width; determining one or more quantization parameter values ​​based on the current tensor; and quantizing the current tensor to a lower bit width based on one or more quantization parameter values ​​determined based on a previous tensor generated during the training of the neural network.

[0010] Some aspects provide a method comprising: collecting activation data statistics when using a trained neural network to generate inference; determining one or more quantization parameter values ​​based on the activation data statistics; and using the one or more quantization parameter values ​​during a refinement operation for the trained neural network.

[0011] Other aspects include: a processing system configured to perform the foregoing methods and those methods described herein; a non-transient computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including means for performing the foregoing methods and those methods further described herein.

[0012] The following description and related figures illustrate certain illustrative features of one or more aspects. Brief description of the attached diagram

[0014] The accompanying drawings depict some of these aspects and are therefore not intended to limit the scope of this disclosure.

[0015] Figure 1 An example workflow for processing data during the forward pass of a quantized machine learning model is described.

[0016] Figure 2 An example workflow for training a quantized machine learning model during backpropagation is described.

[0017] Figure 3 An example workflow is described for using past statistics to inform the current quantization parameters when training a machine learning model.

[0018] Figure 4 Example tensor distributions and quantization saturation are depicted to inform the quantization parameters used to train machine learning models.

[0019] Figure 5 This is an example flowchart illustrating a method for quantizing activation data during the forward pass of a machine learning model.

[0020] Figure 6 This is an example flowchart illustrating a method for quantizing gradient data during the backpropagation of a machine learning model.

[0021] Figure 7 This is an example flowchart illustrating a method for quantized training of machine learning models.

[0022] Figure 8 This is an example flowchart illustrating a method for generating quantitative statistics during inference using machine learning models.

[0023] Figure 9 An example processing system configured to perform various aspects of this disclosure is described.

[0024] To facilitate understanding, the same reference numerals are used wherever possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration.

[0025] Detailed description

[0026] Various aspects of this disclosure provide techniques for using historical tensor statistics in quantized machine learning to improve the efficiency of the training process. As used herein, training can refer to any update or modification of a machine learning model, such as iterative training, intermittent training, continuous training, joint training, etc.

[0027] The aspects of this disclosure can be applied to both forward propagation of data over a network (e.g., during inference or training) and backward propagation (e.g., using backpropagation during training). Implementing quantized machine learning using the aspects of this disclosure achieves a variety of benefits, including reduced computational complexity in processing data, reduced power consumption, reduced latency, and so on.

[0028] The conventional effort required to quantize the training of machine learning models, such as deep neural network models, drastically reduces the efficiency of quantized training. Existing methods of this kind cannot enable training on edge devices without server computing resources. For example, such edge devices are often constrained or limited in terms of available power (e.g., on battery-powered devices), memory, processor speed, availability of computing elements or modules, thermal limitations, etc. Training on such edge devices may be desirable for a variety of reasons, including protecting privacy and enabling personalized AI.

[0029] Quantizing data (e.g., quantizing gradients) during the backpropagation process of training machine learning models can provide considerable speedup and power efficiency, but noise introduced by gradient quantization, if left uncontrolled, can impair the accuracy of the network. In various aspects of this disclosure, by carefully selecting quantization parameters, quantized training can achieve similar or better accuracy than floating-point (FP) training across a range of tasks and models. In some respects, this can be achieved by quantizing data (such as weights, weight gradients, activations, and activation gradients, e.g., quantized to 8 bits) while simultaneously maintaining certain operations (e.g., batch normalization or weight updates) in floating-point (e.g., 16-bit or 32-bit) format.

[0030] Some aspects of this disclosure enable efficient quantized training using defined quantization parameters (e.g., quantization ranges) of gradients, activations, and / or weights. Since these tensors are typically unbounded, appropriately choosing the quantization range can help keep quantization error under control.

[0031] To define these quantization parameters in various aspects of this disclosure, the system may consider various statistical properties of the input data, including the minimum and maximum ranges of the data tensors, the standard deviation of the tensors, saturation of previous quantization, and so on. In some aspects, the system may use a moving average of previous tensor statistics.

[0032] To determine the quantization parameters of an input data tensor, conventional systems require access to the entire unquantized tensor, which is typically unavailable without significant data transfer and memory overhead. That is, since tensors are usually computed piecewise and the quantization range depends on the complete tensor output, conventional systems need to write the entire high-precision tensor to memory piecewise during generation. In other words, because hardware generates only a portion of the tensor at a time, each portion must be sequentially written to memory to enable analysis of the entire tensor.

[0033] In contrast, various aspects of this disclosure utilize hindsight-based quantization range estimation to provide hardware-friendly quantized training. To this end, the system can quantize the current tensor using the quantization range estimated based on previous training iterations. While tensors are used as example data structures for quantization using some aspects of this disclosure, the quantization techniques described herein can be readily applied to any data, regardless of its specific structure or format. This approach enables fast static quantization of gradients and activations. In some aspects of this disclosure, simple hardware support from neural network accelerators is used to track output statistics in an online manner. Furthermore, this allows the processing system to significantly accelerate computation and reduce memory overhead by using pre-computed quantization ranges. In some aspects, a moving average of the quantization range can be used, and this range can be updated efficiently and online based on statistics extracted from the accumulator.

[0034] The quantization techniques discussed in this paper can be applied to achieve fully quantized training of machine learning models (such as deep neural networks) (during both forward and backward passes). These aspects can also be applied to inference using deep neural networks to achieve fast and efficient execution. Even on resource-constrained devices, this enables improved machine learning (e.g., training and inference with reduced computational cost).

[0035] For example, during quantization training, tensor statistics from earlier training iterations can be used when quantizing tensors in the current and / or subsequent iterations. In some respects, once the model is trained, statistics can be collected for the activation tensors generated during inference. These statistics can then be used during subsequent training or refinement operations to determine appropriate quantization parameters when quantizing activation tensors generated during forward passes through the training data.

[0036] Examples during forward propagation are trained by quantization

[0037] Figure 1 An example workflow 100 for processing data during the forward pass of a quantized machine learning model is described.

[0038] In various aspects of this disclosure, neural network training can generally be considered as a series of iterations or steps, where each iteration includes forward passes and backward passes. A forward pass refers to the process in which data is passed from the beginning of the network through each neuron and layer to the end. The loss can then be calculated, and backward passes (e.g., backpropagation) are used to refine the model's weights or other parameters (starting at the last layer and moving towards the first). In some aspects, "forward pass" can also refer to processing the input data during inference.

[0039] In the illustrated workflow 100, a quantized input 105 is received at one layer in the model. This input can be in the form of a tensor received from the previous layer in the model (or, in the case of the first layer, as a direct input to the model). In some respects, input 105 is referred to as activation data or activation tensor. That is, input 105 can be activation data from the previous layer. As used herein, data is “activated data” if it has been processed with an activation function (e.g., in the previous layer), and is “unactivated data” when it has not yet been processed with an activation function (e.g., in the current layer). Although input 105 is depicted as quantized in the illustrated example, in some respects, input 105 can be full precision.

[0040] The weights 110 of each layer can be stored in a high-precision format (e.g., 16 or 32-bit floating-point) to allow for the accumulation of small gradients during training. For the forward pass, the weight tensor is quantized into a lower-bit weight tensor 120 via a weight quantization function 115. This reduces the computational cost of using the weights during the forward pass.

[0041] During workflow 100, the quantized weight tensor 120, along with the quantized input 105, can then be provided to the multiplication and accumulation (MAC) array 125. A MAC array is typically a hardware or software component configured to perform multiplication and accumulation operations, where each value in the two tensors is multiplied together and added to the running sum via an accumulator. For example, a MAC array can compute the dot product or inner product of the input tensors. The MAC array 125 can then compute the linear operations of the layer in a conventional manner, resulting in tensor 130.

[0042] Generally, since the quantized input tensor 105 and quantized weights 120 are typically larger than the MAC array 125, the output tensor 130 is computed through multiple computation loops. The output tensor 130 can then be followed by quantization operation 135 to convert it to the desired bit width. For example, the output tensor 130 can be 32 bits, while the quantized output tensor 140 can be 8 bits.

[0043] As explained, in order to perform quantization 135, the system can use previous statistics 133. These statistics reflect the characteristics of the output tensor 130 during one or more previous iterations of the training process. That is, the system can use the previous statistics 133 of the output tensor generated during one or more previous iterations, rather than evaluating the current statistics of the current output tensor 130, to define the quantization parameters for the current iteration.

[0044] In some respects, the activation function can be applied to the output tensor 130 before quantization operation 135, or to the quantized tensor output 140 after quantization operation 135, or both. That is, depending on the complexity of the activation function, the system can apply the activation function before quantization operation 135, after quantization operation 135, or both before and after quantization operation 135. For example, a modified linear unit (ReLU) activation can be applied after quantization, while other more complex activation functions need to be applied before and after quantization.

[0045] As discussed above, dynamic quantization relies on the statistics of a full-precision tensor to quantize that tensor. For example, to perform quantization operation 115, the statistics of the weight tensor 110 must be known (e.g., the minimum and maximum values ​​of the weight tensor 110, which can be used to define the quantization range of quantization operation 115). In the case of static quantization, the quantization range is known in advance. For example, since the weights are known, the statistics of the weights 110 can also be pre-computed, allowing for efficient execution of quantization operation 115.

[0046] However, for quantization operation 135, the statistics of output tensor 130 are not known until the tensor itself is computed. Generally, the size of the input tensor 105 and weight tensor 120 multiplied in MAC array 125 exceeds the size of the array. For this reason, computation is performed slice by slice until the entire tensor multiplication is complete. The accumulator output typically has a high bit width (e.g., 32 bits) to avoid overflow. That is, since accumulators can typically accumulate large sums (easily exceeding the maximum value that can be represented using the available bits in the input data), the output typically has a high bit width to ensure that the accumulated value does not overflow a smaller bit width.

[0047] To extract the necessary statistics from an existing or conventional system, all outputs of the MAC array 125 or the accumulator associated with the MAC array 125 (e.g., all portions of the output tensor 130) must first be written to memory. In an existing system, after obtaining the complete tensor, statistics can be extracted and the quantization range can be computed. The existing system must then bring the output tensor 130 back to the computation unit (from memory) for quantization using these parameters and then store it back into memory.

[0048] These existing methods result in significant memory overhead and additional data movement. The aspects of this disclosure therefore utilize statistics from previous iterations to control the current quantization. This allows for pre-computed parameters, thereby enabling efficient execution of the quantization operation 135.

[0049] Therefore, during the current training iteration, the system can efficiently extract statistics from the output tensor 130 when generating the output tensor 130 (e.g., without needing to write them to memory). For example, when generating each part of the tensor, the system can identify the minimum and maximum values ​​of that part. This part can then be quantized using quantization operation 135 based on quantization parameters determined during one or more previous iterations.

[0050] Once the entire tensor has been generated and evaluated, the overall minimum and maximum values ​​of tensor 130 during the current iteration can be used to set the quantization range for the next training iteration. During subsequent iterations, these ranges are used to perform quantization operations 135 (while simultaneously collecting updated statistics for subsequent iterations).

[0051] That is, during the first training iteration, a first set of statistics is used to determine a first set of quantization parameters. During the second (subsequent) training iteration, the first set of quantization parameters is used to quantize the tensor, while a second set of tensor statistics is used to determine a second set of quantization parameters (e.g., by refining the first set of quantization parameters based on a moving average). During the third training iteration (after the second iteration), this second set of quantization parameters is used to quantize the data. This approach allows for efficient quantization operations with improved memory overhead and memory transfer, as well as reduced power consumption and latency.

[0052] When data first passes through the model during forward propagation (e.g., during the first training iteration), the quantization parameters may not be available (because there are no previous iterations or statistics). Therefore, in some respects, the system can perform conventional dynamic quantization (e.g., write the output tensor 130 to memory for overall evaluation and use these values ​​to perform the first quantization operation 135). Subsequent iterations can use the post-observation-based techniques described herein. In other respects, the quantization parameters used for this first round can be determined using other techniques (such as randomly or pseudo-randomly, heuristically, etc.).

[0053] Examples during backpropagation are trained by quantization

[0054] Figure 2 An example workflow 200 for training a quantized machine learning model during backpropagation is described.

[0055] During the backpropagation, the quantized activation gradient 205 is used to compute the weight gradient 220 and the input gradient 255. Specifically, the MAC array 215 is used to compute the weight gradient 220 based on the quantized activation gradient 205 and the quantized input tensor 210. In some respects, the quantized input tensor 210 corresponds to the quantized input tensor 105 used in the forward propagation.

[0056] In some respects, the weight gradient 220 can be quantized into a lower-bit representation using quantization operation 225 (e.g., quantized weight gradient 230). In other respects, the weight gradient 220 can be kept in full precision. In the explained workflow 200, the quantized weight gradient 230 is used by optimizer 235 to refine the weights 240 of the layer.

[0057] In accordance with the above reference Figure 1 Similar to the quantization operation 135 discussed, quantization operation 225 can be performed using quantization parameters determined based on statistics of the weight gradient 220 in previous iterations (e.g., previous statistics 223). That is, current statistics of the weight gradient 220 can be collected for future iterations, while using previous statistics to perform the current quantization operation 225.

[0058] In the explained workflow 200, the input gradient 255 is also computed by the MAC array 250 based on the quantized activation gradient 205 and the quantized weight tensor 245. Since the input gradient 255 may be quite large, it can be quantized into a lower bit representation (e.g., quantized input tensor 265) using quantization operation 260 before propagating the input gradient 255 to the previous layer.

[0059] In accordance with the above reference Figure 1 Similar to quantization operations 135 and 225, quantization operation 260 can be performed using quantization parameters determined based on statistics (e.g., previous statistics 257) from the input gradient 255 of previous iterations. That is, current statistics of the input gradient 255 can be collected for future iterations, while using previous statistics to perform the current quantization operation 260.

[0060] As discussed above, this use of historical statistics from previous iterations enables the efficient execution of quantization operations 225 and 260, thereby reducing memory and computational overhead, as well as power consumption and latency.

[0061] When data first passes through the model during backpropagation (e.g., during the first training iteration), the quantization parameters may not be available (because there are no previous iterations or statistics). Therefore, in some respects, the system can perform conventional dynamic quantization (e.g., writing the weight tensor 220 and / or the input tensor 255 to memory for overall evaluation and using these values ​​to perform the first quantization operations 225 and 260). Subsequent iterations can use techniques based on hindsight, as described herein (and explained via previous statistics 223 and 257). In other respects, the quantization parameters used for this first round can be determined using other techniques (such as randomly or pseudo-randomly, heuristically, etc.).

[0062] Example quantization parameters used for quantization training are determined.

[0063] Figure 3 An example workflow 300 is described for using past statistics to inform the current quantization parameters when training a machine learning model. Although the illustrated workflow 300 uses gradient quantization during backpropagation as an example, aspects of this disclosure can be readily applied to any data quantized during machine learning, including activation quantization, weight quantization, and so on.

[0064] In the explained workflow 300, the quantized activation gradient 305 and the quantized weight tensor 310 are processed using a MAC array 315. In some respects, the quantized activation gradient 305 and the quantized weight tensor 310 correspond to the referenced above. Figure 2 The quantized activation gradient 205 and the quantized weight tensor 245 are discussed.

[0065] MAC array 315, in conjunction with accumulator 320 (which may be a discrete component or part of MAC array 315), generates input gradient 325 based on quantized activation gradient 305 and quantized weight tensor 310. The accumulator typically accumulates the sum of products computed by MAC array 315. As explained, statistics of input gradient 325 are collected by accumulator 320 and output to block 330, which generates or updates the set of quantization parameters 335. For example, as accumulator 320 outputs portions of input gradient 325, the system can determine the minimum, maximum, standard deviation, etc., of the input gradient 325 (parts). In the explained aspect, quantization parameters 335 are minimum and maximum quantized values ​​(e.g., quantization range). In one aspect, the system can set the minimum and maximum quantized values ​​to be equal to the minimum and maximum tensor values ​​of the current tensor.

[0066] As explained, during the current iteration at time t, the current set of quantization parameters 335 is determined based on statistics of the current input gradient 325. While these quantization parameters 335 are determined at time t based on the current tensor, they will be used to quantize the input gradient during subsequent iterations at time t+1. When computing the MAC output (input gradient 325), the system tracks, for example, minimum-maximum statistics from accumulator 320. Other statistics may be used in other examples. These statistics are used to update the quantization range once the complete tensor is computed (e.g., once the final portion of the tensor is generated). For example, block 330 may track the lowest and highest values ​​observed in any completed portion of the tensor. When the tensor is complete, block 330 can use these values ​​as quantization parameters 335.

[0067] Furthermore, in order to quantize the current tensor at the current t-th iteration, the system uses a quantization range determined based on one or more previous iterations. As explained, during this t-th iteration, the system uses a quantization parameter of 340 (labeled q). t However, these are collected at time t-1) to perform quantization 345 to generate a quantized input gradient 350, which is propagated to the previous layer of the model. The quantization parameter 340 is defined at least in part based on one or more previous iterations. As explained, the quantization parameter 340 used for quantization 345 at the current iteration is not defined based on the current tensor statistics. That is, the quantization parameter 340 is defined only based on statistics from one or more previous iterations, because the current statistics of the input gradient 325 may not be known until quantization 345 begins.

[0068] In some respects, the quantization parameter 335 is defined as the minimum and maximum values ​​of the input gradient 325 (as in the illustrated example). In other respects, the quantization parameter is defined as a moving average of the parameter values. For example, in one such respect, the quantization parameter (defined as q) used at time t (and collected at time t-1) t ) is defined using Equation 1 below, where η is the momentum term and G t-1 The input gradient at time t-1 is 325, q t-1 It is the quantization parameter at time t-1, and stat(·) is a statistical operation (e.g., minimum, maximum, etc.).

[0069] q t =(1-η)stat(G t-1 )+ηq t-1 (1)

[0070] For example, the minimum and maximum values ​​of the quantization range at time t (respectively...) and ) can be defined as:

[0071] and

[0072] In some respects, during the first iteration (at time t = 0), the quantization parameter set q 0 It can be defined as stat(G) 0 As discussed above, during the first iteration, the statistics of the current tensor can be used to define the quantization parameters. In subsequent iterations, the statistics of one or more previous tensors are used to define the quantization parameters.

[0073] Example quantization parameter update based on quantization saturation

[0074] Figure 4 An example tensor distribution 400 and quantization saturation are depicted to inform the quantization parameters used to train a machine learning model.

[0075] Distribution 400 describes the distribution of values ​​in a tensor (e.g., the gradient tensor). In the described distribution 400, different data values ​​are mapped to the horizontal axis, and the frequency of that data value in the tensor is reflected via the vertical axis. Thus, for example, values ​​near the middle of the distribution are frequently represented by the tensor, while values ​​near either extreme are not frequently represented by the tensor. In some respects, tensor values ​​generally form an approximately normal distribution. In the described aspect, a minimum value (indicated by line 405) and a maximum value (indicated by line 410) have been used to define the quantization range. That is, when quantizing the tensor, values ​​in range 415 are within the quantization range, while values ​​outside this range (in ranges 420 and 425) are clipped. These values ​​in ranges 420 and 425 can be referred to as saturated.

[0076] The saturation ratio α can be defined as the proportion of values ​​clipped during quantization. In one aspect, the saturation ratio is defined using Equation 2, where G is a tensor and q... min It is the minimum value of the quantization range, q max It is the maximum value of the quantization range, and |·| represents the number of elements in the tensor.

[0077]

[0078] In some respects, this saturation ratio can be used to define the quantization range for subsequent iterations. For example, in one aspect, if the saturation ratio of the current iteration is within a range defined by a minimum and a maximum saturation threshold (e.g., saturation ratio C...), then... min or α min and C max or α maxWithin the acceptable range defined by q, the quantization range will not be changed or updated. If these limits are exceeded, the quantization range for the next iteration can be updated. In one such aspect, it can be determined according to q. t =λq t To update the quantization range. λ can be defined using Equation 3, where CDF -1 It is the inverse cumulative distribution function, α is the saturation ratio, and step is the defined step size. The step size can be a defined value (e.g., a hyperparameter) used to increase or decrease the saturation ratio.

[0079]

[0080] In at least one aspect, the quantization range is updated only when the saturation ratio consistently falls outside the defined saturation range (e.g., for at least τ iterations), rather than updating the quantization range every time the saturation ratio falls outside the range.

[0081] In the illustrated example, the system has updated the quantization ranges (indicated by arrows 407 and 412) to new minimum and maximum values ​​(indicated by lines 430 and 440, respectively) for subsequent quantization. This results in the smaller portions of the tensor (indicated by ranges 445 and 450) being truncated.

[0082] Example methods for quantizing activation data during the forward pass of a quantized machine learning model.

[0083] Figure 5 This is an example flowchart illustrating method 500 for quantizing activation data during the forward pass of a machine learning model.

[0084] Method 500 begins at box 505, where an input tensor is received. For example, activation tensors (which may be quantized or unquantized) may be received from previous layers in the model. In at least one aspect, the received input tensor may correspond to a reference tensor. Figure 1 The quantized input tensor under discussion is 10⁵.

[0085] In box 510, the output tensor is computed based on the received input tensor and weight tensor. For example, as... Figure 1 As explained, the output tensor 130 (which can be either pre-activation data or activation data) can be computed by multiplying the input tensor and the weight tensor. In some respects, this output tensor is computed using a fixed-point format with a relatively large bit width (e.g., 32 bits). In other respects, to improve the efficiency of the training process, the output tensor can therefore be quantized to a lower bit width.

[0086] In box 515, the output tensor is quantized based on one or more previous statistics of the layer. For example, as discussed above, statistics or properties of the output tensor during one or more previous iterations can be collected and evaluated to generate quantization parameters (e.g., quantization range) for the current quantization. It is important to note that the statistics or properties of the current output tensor are not considered when determining the quantization parameters for the current iteration. That is, the current output tensor is quantized solely based on previous statistics and is not evaluated or analyzed to determine the current quantization parameters. However, the current statistics of the current tensor can be used to update the quantization parameters that will be used in one or more subsequent iterations.

[0087] In some respects, the individual parts of the output tensor are generated separately. However, since the system defines the quantization parameters based on previous iterations, these parts can be quantized individually immediately (instead of waiting to generate the entire tensor).

[0088] In box 520, the statistics of the current output tensor (generated in box 510) are determined. This may include, for example, determining the minimum and / or maximum values ​​of the tensor. In some aspects, the standard deviation of the current tensor is determined. In at least one aspect, the saturation ratio of the current quantization is determined (performed in box 515).

[0089] In box 525, the quantization parameters are updated based on the current statistics (determined in box 520). These updated quantization parameters can then be used in one or more subsequent iterations.

[0090] In some respects, a moving average of the previous parameters is used to update the quantization parameters. For example, Equation 1 above can be used to update the minimum and / or maximum values ​​of the quantization range. As another example, Equation 3 above can be used to update the minimum and / or maximum values ​​of the quantization range based on the saturation ratio.

[0091] These updated quantization parameters can then be used in subsequent training iterations.

[0092] Example methods for quantizing gradient data during backpropagation of a quantized machine learning model.

[0093] Figure 6 This is an example flowchart illustrating method 600 for quantizing gradient data during the backpropagation of a machine learning model.

[0094] Method 600 begins at box 605, where a tensor of activation gradients is received. For example, the activation gradient tensor (which may be quantized or unquantized) may be received from subsequent layers in the model during backpropagation. In at least one aspect, the received activation gradient tensor may correspond to a reference... Figure 2 The activation gradient under discussion is 205.

[0095] In box 610, a gradient tensor is computed based on the received activation gradient tensor and weight tensor. In some respects, this gradient tensor can be either the input gradient tensor or a tensor of the weight gradients. For example, as... Figure 2 As explained in the text, the input gradient 255 can be calculated by multiplying the activation gradient tensor and the weight tensor. Similarly, as... Figure 2 As explained in the text, the weight gradient 220 can be calculated by multiplying the activation gradient tensor 205 and the input tensor 210.

[0096] In some respects, the gradient tensor is computed using a fixed-point format with a relatively large bit width (e.g., 32 bits). In other respects, to improve the efficiency of the training process, the input tensor can therefore be quantized to a lower bit width. In some respects, if the gradient tensor is a weighted gradient tensor, then the weighted gradient tensor can be kept at a high bit width instead of being quantized.

[0097] In box 615, the gradient tensor is quantized based on one or more previous statistics of the layer. For example, as discussed above, statistics or properties of the gradient tensor during one or more previous iterations can be collected and evaluated to generate quantization parameters (e.g., quantization range) for the current quantization. It is important to note that the statistics or properties of the current gradient tensor are not considered when determining the quantization parameters for the current iteration. That is, the current gradient tensor is quantized solely based on previous statistics and is not evaluated or analyzed to determine the current quantization parameters. The current statistics of the current gradient tensor can be used to update the quantization parameters that will be used in one or more subsequent iterations.

[0098] In some respects, the individual parts of the gradient tensor are generated separately. However, since the system defines the quantization parameters based on previous iterations, these parts can be quantized individually immediately (instead of waiting to generate the entire tensor).

[0099] In box 620, the statistics of the current gradient tensor (generated in box 610) are determined. This may include, for example, determining the minimum and / or maximum values ​​of the tensor. In some aspects, the standard deviation of the current tensor is determined. In at least one aspect, the saturation ratio of the current quantization is determined (performed in box 615).

[0100] In box 625, the quantization parameters are updated based on the current statistics (determined in box 620). These updated quantization parameters can then be used in one or more subsequent iterations.

[0101] In some respects, a moving average of the previous parameters is used to update the quantization parameters. For example, Equation 1 above can be used to update the minimum and / or maximum values ​​of the quantization range. As another example, Equation 3 above can be used to update the minimum and / or maximum values ​​of the quantization range based on the saturation ratio.

[0102] These updated quantization parameters can then be used in subsequent training iterations.

[0103] Example methods for quantized training of machine learning models

[0104] Figure 7 This is an example flowchart illustrating method 700 for quantized training of machine learning models.

[0105] Method 700 begins at box 705, where the current tensor is generated with the first width. In some respects, the current tensor is a gradient tensor. In some respects, the current tensor is an activation tensor. In some respects, the current tensor is generated based on the tensors received at a layer of the neural network.

[0106] In box 710, one or more quantization parameter values ​​are determined based on the current tensor. In some aspects, the one or more quantization parameter values ​​determined based on the current tensor are calculated based on the minimum and maximum current tensor values. In other aspects, the one or more quantization parameter values ​​determined based on the current tensor are calculated based on the standard deviation of the current tensor values.

[0107] In some respects, determining the value of one or more quantization parameters based on the current tensor includes: a moving average based on the current tensor and previous tensors to determine the value of one or more quantization parameters.

[0108] In some respects, the current minimum quantization parameter value according to Calculated based on the previous tensor T t-1 and the previous minimum quantization parameter value The moving average, and the current maximum quantization parameter value. according to Calculated based on the previous tensor T t-1 and the previous maximum quantization parameter value The moving average of η, where η is the momentum parameter.

[0109] In some respects, the one or more quantization parameter values ​​determined based on the current tensor include the quantization range calculated based on the saturation ratio used to quantize the current tensor.

[0110] In some respects, the quantization range is updated based on one or more thresholds associated with the saturation ratio.

[0111] In some respects, the one or more thresholds include the maximum saturation threshold C. max and minimum saturation threshold C min The quantization range is determined by the current quantization parameter value q. t Define and update the quantization range according to the following formula:

[0112] q t =λq t ;in

[0113] CDF -1 It is the inverse cumulative distribution function, α is the saturation ratio, and step is the defined step size.

[0114] In some aspects, method 700 further includes: storing the one or more quantization parameter values ​​determined based on the current tensor in memory for use in subsequent tensor quantization, and quantizing subsequent tensors based on the one or more quantization parameter values ​​determined based on the current tensor during subsequent passes of data through layers of the neural network.

[0115] In box 715, the current tensor is quantized to a lower bit width based on one or more quantization parameter values ​​determined based on previous tensors generated during the training of the neural network. In some respects, using these one or more quantization parameters determined based on previous tensors reduces the memory passes required to quantize the current tensor.

[0116] Example methods for generating quantitative statistics during inference using machine learning models.

[0117] Figure 8 This is an example flowchart illustrating a method 800 for generating quantitative statistics during inference using a machine learning model.

[0118] Method 800 begins at box 805, where activation data statistics are collected while generating inference using a trained neural network.

[0119] In box 810, one or more quantization parameter values ​​are determined based on the activation data statistics.

[0120] In some respects, the values ​​of one or more quantized parameters determined based on activation data statistics are calculated based on the minimum and maximum values ​​of the activation data.

[0121] In some respects, the values ​​of one or more quantized parameters determined based on activation data statistics are calculated based on the standard deviation of the activation data.

[0122] In some respects, the values ​​of one or more quantization parameters determined based on activation data statistics include a quantization range calculated based on the saturation ratio used to quantize the activation data.

[0123] In some respects, the quantization range is updated based on one or more thresholds associated with the saturation ratio.

[0124] In box 815, one or more quantization parameter values ​​are used during the refinement operation for the trained neural network.

[0125] In some aspects, method 800 further includes, during the refinement operation: generating a current tensor with a first bit width; determining one or more quantization parameter values ​​based on the current tensor; and quantizing the current tensor to a lower bit width based on one or more quantization parameter values ​​determined based on activation data statistics.

[0126] Example processing system for quantized training

[0127] In some respects, regarding Figure 1-8 The described techniques, methods, and workflows can be implemented on one or more devices or systems.

[0128] Figure 9 An example processing system 900 is depicted, which can be configured to perform aspects of the various methods described herein, including, for example, those concerning... Figure 1-9 The aspects described.

[0129] The processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed at the CPU 902 may be loaded, for example, from program memory associated with the CPU 902 or from memory partition 924.

[0130] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, a multimedia processing unit 910, and a wireless connectivity component 912.

[0131] NPUs (such as the 908) are generally dedicated circuits configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0132] NPUs (such as the 908) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, multiple NPUs may be part of a dedicated neural network accelerator.

[0133] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0134] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0135] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and rapidly process them through an already trained model to generate model outputs (e.g., inference).

[0136] In one implementation, the NPU 908 is part of one or more of the CPU 902, GPU 904, and / or DSP 906.

[0137] In some examples, the wireless connectivity component 912 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 912 is further connected to one or more antennas 914.

[0138] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0139] The processing system 900 may also include one or more input and / or output devices 922, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0140] In some examples, one or more processors of the processing system 900 may be based on the ARM or RISC-V instruction set.

[0141] The processing system 900 also includes a memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 924 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 900.

[0142] Specifically, in this example, the memory 924 includes a set of computational components 924A, statistical components 924B, quantization parameter components 924C, quantization components 924D, quantization parameters 924E, a training component 924F, an inference component 924G, and a set of model parameters 924H. These components may be configured according to one or more aspects described herein.

[0143] For example, computation component 924A can be configured to generate activation tensors, gradient tensors, weight tensors, etc., during training and / or inference (e.g., using MAC circuits 926).

[0144] The statistics component 924B can be configured to extract statistics from the generated tensors, such as minimum, maximum, standard deviation, quantization saturation ratio, etc.

[0145] The quantization parameter component 924C can be configured to update the quantization parameter 924E based on the current statistics in each iteration of the training process.

[0146] The quantization component 924D can be configured to quantize the generated tensor based on the quantization parameter 924E, which is generated or updated based on previous training iterations.

[0147] The training component 924F and the inference component 924G can generally be configured to: train one or more models (e.g., to refine the set of model parameters 924H) and use the models (e.g., using the trained model parameters 924H) to generate inference. For example, the training component 924F and the inference component 924G can be trained and use quantized machine learning techniques, such as... Figure 1-8 The quantitative machine learning techniques explained in the article.

[0148] The set of model parameters 924H can generally include parameters used for one or more machine learning models (e.g., neural networks) (including the quantized models discussed in this paper).

[0149] The processing system 900 further includes one or more MAC circuits 926 and quantization circuits 928, which can be configured to perform multiplication-accumulation operations and quantization operations respectively via hardware.

[0150] Although Figure 9For clarity, they are depicted as separate circuits, but the MAC circuits 926 and quantization circuits 928 can be implemented in other processing devices of the processing system 900 (such as CPU 902, GPU 904, DSP 906, NPU 908, etc.).

[0151] Generally, the processing system 900 and / or its components can be configured to perform the methods described herein.

[0152] It is worth noting that, in other aspects, various features of the processing system 900 may be omitted, such as when the processing system 900 is a server computer. For example, in other aspects, the multimedia component 910, wireless connectivity 912, sensor 916, ISP 918, and / or navigation component 920 may be omitted. Furthermore, various features of the processing system 900 may be distributed among multiple devices.

[0153] The described components, as well as other undescribed components, can be configured to perform various aspects of the methods described herein.

[0154] Example Terms

[0155] Clause 1: A method for training a neural network model, comprising: receiving a tensor at a layer of the neural network; generating a current tensor with a first bit width based on the received tensor; determining one or more quantization parameter values ​​based on the current tensor; and quantizing the current tensor to a lower bit width according to one or more quantization parameter values ​​determined based on a previous tensor generated during training of the neural network, wherein using the one or more quantization parameters determined based on the previous tensor reduces the memory passes required to quantize the current tensor.

[0156] Clause 2: According to the method of Clause 1, wherein the one or more quantization parameter values ​​determined based on the current tensor are calculated based on the minimum current tensor value and the maximum current tensor value.

[0157] Clause 3: The method of any of Clauses 1-2, wherein the value of the one or more quantization parameters determined based on the current tensor is calculated based on the standard deviation of the current tensor value.

[0158] Clause 4: The method according to any one of Clauses 1-3 further includes: storing the one or more quantization parameter values ​​determined based on the current tensor in memory for use in subsequent tensor quantization; and quantizing subsequent tensors based on the one or more quantization parameter values ​​determined based on the current tensor during subsequent propagation of data through the layer of the neural network.

[0159] Clause 5: The method of any of Clauses 1-4, wherein determining the value of the one or more quantization parameters based on the current tensor includes: a moving average of the values ​​of the one or more quantization parameters based on the current tensor and the previous tensor.

[0160] Clause 6: According to the method of any of Clauses 1-5, where: the current minimum quantization parameter value according to Calculated based on the previous tensor T t-1 and the previous minimum quantization parameter value The moving average, and the current maximum quantization parameter value. according to Calculated based on the previous tensor T t-1 and the previous maximum quantization parameter value The moving average of η, where η is the momentum parameter.

[0161] Clause 7: According to the method of any of Clauses 1-6, wherein the one or more quantization parameter values ​​determined based on the current tensor include a quantization range calculated based on the saturation ratio used to quantize the current tensor.

[0162] Clause 8: The method of any of Clauses 1-7, wherein the quantization range is updated based on one or more thresholds associated with the saturation ratio.

[0163] Clause 9: According to the method of any one of Clauses 1-8, wherein: the one or more thresholds include the maximum saturation threshold C. max and minimum saturation threshold C min The quantization range is determined by the current quantization parameter value q. t It is defined by q, and the quantization range is updated based on q. t =λq t To be executed; among them CDF -1 It is the inverse cumulative distribution function, α is the saturation ratio, and step is the defined step size.

[0164] Clause 10: The method of any of Clauses 1-9, wherein the current tensor is a gradient tensor.

[0165] Clause 11: The method of any of Clauses 1-10, wherein the current tensor is the active tensor.

[0166] Clause 12: A method comprising: collecting activation data statistics while using a trained neural network to generate inference; determining one or more quantization parameter values ​​based on the activation data statistics; and using the one or more quantization parameter values ​​during a refinement operation for the trained neural network.

[0167] Clause 13: According to the method of Clause 12, wherein the one or more quantized parameter values ​​determined based on the activation data statistics are calculated based on the minimum and maximum values ​​of the activation data.

[0168] Clause 14: The method of any of Clauses 12-13, wherein the value of the one or more quantized parameters determined based on the activation data statistics is calculated based on the standard deviation of the activation data.

[0169] Clause 15: The method of any of Clauses 12-14, wherein the one or more quantization parameter values ​​determined based on the activation data statistics include a quantization range calculated based on the saturation ratio used to quantize the activation data.

[0170] Clause 16: The method of any of Clauses 12-15, wherein the quantization range is updated based on one or more thresholds associated with the saturation ratio.

[0171] Clause 17: The method according to any one of Clauses 12-16 further includes, during the refinement operation: generating a current tensor with a first bit width; determining one or more quantization parameter values ​​based on the current tensor; and quantizing the current tensor to a lower bit width based on the one or more quantization parameter values ​​determined based on the activation data statistics.

[0172] Clause 18: A system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method pursuant to any of Clauses 1-17.

[0173] Clause 19: A system comprising means for performing the method according to any one of Clauses 1-17.

[0174] Clause 20: A non-transient computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of Clauses 1-17.

[0175] Clause 21: A computer program product contained on a computer-readable storage medium, comprising code for performing a method pursuant to any of Clauses 1-17.

[0176] Additional considerations

[0177] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not intended to limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements in discussion may be altered without departing from the scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice a method. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or differ from the aspects set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be implemented by one or more elements of the claims.

[0178] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” should not be construed as superior to or better than the others.

[0179] As used herein, the phrase “at least one of” a list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0180] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and the like. Moreover, "determine" can include parsing, selecting, choosing, building, and the like.

[0181] As used herein, in the context of sharing electronic signals and data among the elements described herein, the term "connected to" generally refers to data communication between corresponding elements that are connected to each other. In some cases, elements may be directly connected to each other, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and / or data between corresponding elements that are directly connected to each other. In other cases, elements may be indirectly connected to each other, such as via one or more data buses or similar shared circuit systems and / or integrated circuit elements used to transmit signals and data between corresponding elements that are indirectly connected to each other.

[0182] The methods disclosed herein include one or more steps or actions for implementing the method. These method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the drawings are present, these operations may have corresponding paired means with similar numbers plus functional components.

[0183] The following claims are not intended to be limited to the aspects shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean “one and only one” (unless specifically stated so), but rather “one or more.” Unless specifically stated otherwise, the term “some / a” refers to one or more. No element of the claims should be interpreted in accordance with the provisions of 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “steps for…”. Elements of all aspects described throughout this disclosure that are now or hereafter known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is expressly stated in the claims.

Claims

1. A computer-implemented method for updating a neural network and performing machine learning tasks, comprising: Tensors are received at one layer of the neural network; Generate the current tensor with the first width based on the received tensor; Determine one or more quantization parameter values ​​based on the current tensor; The current tensor is quantized to a lower bit width based on one or more quantization parameter values ​​determined based on a previous tensor generated during the update of the neural network, wherein using the one or more quantization parameters determined based on the previous tensor reduces the memory passes required to quantize the current tensor. Receive sensor data for the machine learning task; as well as Performing the machine learning task includes outputting the result of the machine learning task, wherein the machine learning task includes image classification, machine translation, or object detection. Specifically, determining the one or more quantization parameter values ​​based on the current tensor includes: determining a moving average of the one or more quantization parameter values ​​based on the current tensor and the previous tensor.

2. The method as described in claim 1, wherein, The one or more quantization parameter values ​​determined based on the current tensor are calculated based on the minimum current tensor value and the maximum current tensor value.

3. The method as described in claim 1, wherein, The one or more quantization parameter values ​​determined based on the current tensor are calculated based on the standard deviation of the current tensor values.

4. The method of claim 1, further comprising: The values ​​of one or more quantization parameters determined based on the current tensor are stored in memory for use in subsequent tensor quantizations; as well as Subsequent tensors are quantized based on the values ​​of one or more quantization parameters determined based on the current tensor, during subsequent propagation of data through the layers of the neural network.

5. The method of claim 1, wherein: Current minimum quantization parameter value according to Computed based on previous tensors and the previous minimum quantization parameter value The moving average, and Current maximum quantization parameter value according to Calculated based on the previous tensor and the previous maximum quantization parameter value The moving average, where It is a momentum parameter.

6. The method of claim 1, wherein, The one or more quantization parameter values ​​determined based on the current tensor include a quantization range calculated based on the saturation ratio used to quantize the current tensor.

7. The method of claim 6, wherein, The quantization range is updated based on one or more thresholds associated with the saturation ratio.

8. The method of claim 7, wherein: The one or more thresholds include the maximum saturation threshold. and minimum saturation threshold , The quantization range is determined by the current quantization parameter value. Defined by, and The quantization range is updated based on To be carried out, in , And among them It is the inverse cumulative distribution function. It is the saturation ratio, and It is a defined step size value.

9. The method of claim 1, wherein, The current tensor is a gradient tensor.

10. The method of claim 1, wherein, The current tensor is the active tensor.

11. A computer-implemented method, comprising: Activation data statistics are collected when using a trained neural network to generate inferences; One or more quantization parameter values ​​are determined based on the activation data statistics; as well as Using the one or more quantization parameter values ​​during a refinement operation for the trained neural network includes performing the steps of any one of claims 1 to 10 during the refinement operation, wherein quantizing the current tensor to a lower bit width is based on the one or more quantization parameter values ​​determined based on the activation data statistics.

12. The method of claim 11, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics are calculated based on the minimum and maximum values ​​of the activation data.

13. The method of claim 11, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics are calculated based on the standard deviation of the activation data.

14. The method of claim 11, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics include a quantization range calculated based on the saturation ratio used to quantize the activation data.

15. The method of claim 14, wherein, The quantization range is updated based on one or more thresholds associated with the saturation ratio.

16. The method of claim 11, further comprising, during the refining operation: Generate the current tensor using the first width; Determine one or more quantization parameter values ​​based on the current tensor; as well as The current tensor is quantized to a lower bit width based on one or more quantization parameter values ​​determined based on the activation data statistics.

17. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and cause the processing system to perform operations, the operations including: Tensors are received at one layer of the neural network; Generate the current tensor with the first width based on the received tensor; Determine one or more quantization parameter values ​​based on the current tensor; The current tensor is quantized to a lower bit width based on one or more quantization parameter values ​​determined based on a previous tensor generated during the update of the neural network, wherein using the one or more quantization parameters determined based on the previous tensor reduces the memory passes required to quantize the current tensor. Receive sensor data for machine learning tasks; and Performing the machine learning task includes outputting the result of the machine learning task, wherein the machine learning task includes image classification, machine translation, or object detection. Specifically, determining the one or more quantization parameter values ​​based on the current tensor includes: determining a moving average of the one or more quantization parameter values ​​based on the current tensor and the previous tensor.

18. The processing system of claim 17, wherein, The one or more quantization parameter values ​​determined based on the current tensor are calculated based on the minimum current tensor value and the maximum current tensor value.

19. The processing system of claim 17, wherein, The one or more quantization parameter values ​​determined based on the current tensor are calculated based on the standard deviation of the current tensor values.

20. The processing system of claim 17, wherein the operation further comprises: The values ​​of one or more quantization parameters determined based on the current tensor are stored in memory for use in subsequent tensor quantizations; as well as Subsequent tensors are quantized based on the values ​​of one or more quantization parameters determined based on the current tensor, during subsequent propagation of data through the layers of the neural network.

21. The processing system of claim 17, wherein, The one or more quantization parameter values ​​determined based on the current tensor include a quantization range calculated based on the saturation ratio used to quantize the current tensor.

22. The processing system of claim 21, wherein, The quantization range is updated based on one or more thresholds associated with the saturation ratio.

23. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and cause the processing system to perform operations, the operations including: Activation data statistics are collected when using a trained neural network to generate inferences; One or more quantization parameter values ​​are determined based on the activation data statistics; and Using the one or more quantization parameter values ​​during a refinement operation for the trained neural network includes performing the steps of any one of claims 1 to 10 during the refinement operation, wherein quantizing the current tensor to a lower bit width is based on the one or more quantization parameter values ​​determined based on the activation data statistics.

24. The processing system of claim 23, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics are calculated based on the minimum and maximum values ​​of the activation data.

25. The processing system of claim 23, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics are calculated based on the standard deviation of the activation data.

26. The processing system of claim 23, wherein, The one or more quantization parameter values ​​determined based on the activation data statistics include a quantization range calculated based on the saturation ratio used to quantize the activation data.

27. The processing system of claim 26, wherein, The quantization range is updated based on one or more thresholds associated with the saturation ratio.

28. The processing system of claim 23, wherein the operation further comprises, during the refining operation: Generate the current tensor using the first width; Determine one or more quantization parameter values ​​based on the current tensor; as well as The current tensor is quantized to a lower bit width based on one or more quantization parameter values ​​determined based on the activation data statistics.

Images