Mixed-precision quantization in machine learning using model sensitivity and constrained optimization

Abstract

Certain aspects of the present disclosure provide techniques for performing mixed precision quantization of a machine learning model. In one example, a method includes determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

Description

MIXED-PRECISION QUANTIZATION IN MACHINE LEARNING USING MODEL SENSITIVITY AND CONSTRAINED OPTIMIZATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This Application claims priority to Greek Application No. 20230100199, filed on March 8, 2023, the entire contents of which is incorporated herein by reference.INTRODUCTIONField of the Disclosure

[0002] Aspects of the present disclosure relate to machine learning, and more particularly, to techniques for performing mixed-precision quantization using model sensitivity and constrained optimization.Description of Related Art

[0003] Machine learning generally refers to methods of training models to perform useful tasks. One type of machine learning model that has proven extremely versatile is the artificial neural network model, or neural network for short.

[0004] A neural network is generally a collection of nodes connected by weighted edges that transmit data signals between the nodes. The nodes are commonly organized into layers, with one layer’s output (e.g., a lower layer) feeding the next layer’s input (e.g., a higher layer). Different layers may generally be configured to perform different types of transformations on their inputs, such as convolutional transformations. A node in a layer may receive one or more data signals from inbound edges connected to other nodes, and those data signals may be adjusted by the edge weights. Further, a node may have a bias as an independent input data signal. The node processes all of the input data signals, for example, with a linear or non-linear function and then “activates” based on the output of the function. In some cases, the activation of a node may cause the transmission of further data signals to further connected nodes. The output of the neural network is often referred to as an inference, and can take many forms, such as a numerical output, a classification output, and others. Training a neural network, often referred to as deep learning, generally involves adjusting the values of the edge weights and biases until the output of the neural network meets some task performance objective.

[0005] While neural networks are powerful machine learning model architectures capable of a wide range of useful tasks, such as recognizing objects in image data, theyare likewise highly resource dependent. For example, neural networks may require significant compute, memory, power, and time resources for training and / or for inferencing. These resource requirements may significantly limit the ability to train and deploy neural networks to certain types of devices and for certain use cases. Thus, there is typically a significant trade-off between neural network task performance and resource usage associated with training and using the neural network. Accordingly, there is a need in the art to improve the efficiency of neural network model architectures.SUMMARY

[0006] Some aspects provide a method for performing mixed precision quantization of a machine learning model. The method includes determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

[0007] Other aspects provide: an apparatus operable, configured, or otherwise adapted to perform any one or more of the aforementioned methods and / or those described elsewhere herein; a non-transitory, computer-readable media comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform the aforementioned methods as well as those described elsewhere herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods as well as those described elsewhere herein; and / or an apparatus comprising means for performing the aforementioned methods as well as those described elsewhere herein. By way of example, an apparatus may comprise a processing system, a device with a processing system, or processing systems cooperating over one or more networks.

[0008] The following description and the appended figures set forth certain features for purposes of illustration.BRIEF DESCRIPTION OF DRAWINGS

[0009] The appended figures depict certain features of the various aspects described herein and are not to be considered limiting of the scope of this disclosure.

[0010] FIG. 1 depicts an example architecture for performing mixed precision neural network quantization.

[0011] FIG. 2 depicts a mixed precision neural network quantization flow.

[0012] FIG. 3 depicts an example method of performing mixed precision quantization.

[0013] FIG. 4 depicts aspects of an example processing system.DETAILED DESCRIPTION

[0014] Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums for performing mixed-precision quantization using model element sensitivity and constrained optimization.

[0015] As above, there is a need in the art to reduce the trade-off between neural network task performance and resource usage (e.g., compute, memory, energy, time, etc.). One such method is quantization, which is generally a method of mapping continuous infinite values to a smaller set of discrete finite values. In other words, quantization approximates real -world values (e.g., floating point values) with representative values (e.g., integer values) that limit the precision and range of the original input. When applied to machine learning, such as to a neural network, quantization may significantly reduce resource usage for both training and inferencing. For example, performing massive numbers of integer operations during training of or inferencing with a quantized (e.g., reduced-precision) neural network may be significantly more efficient in terms of resource usage as compared to performing floating point operations with an unquantized (e.g., full -preci si on) neural network processing the same input data.

[0016] Not all elements of a neural network are equally resilient to quantization. That is, certain elements of a neural network may be more sensitive to quantization than others. Consequently, quantizing an entire neural network to a uniform bitwidth (referred to a fixed-precision quantization (FPQ)) may result in reduced resource usage, but may also reduce task performance to an unacceptable level. To resolve this issue, mixed-precision quantization (MPQ) seeks to quantize different elements of a neural network at varyingquantization rates, which results in reduced resource usage while maintaining task performance. The various elements of a neural network that may be quantized with MPQ include, for example, individual layers, groups of layers, sub-layers, weight channels, and others. Generally, MPQ can achieve higher accuracy for the same computational budget because MPQ allocates higher precision to elements (e.g., layers) that are more sensitive to quantization and reduces bitwidth for elements less sensitive to quantization.

[0017] However, finding the optimal bitwidth allocation for a specific neural network is a challenging combinatorial search problem because the bitwidth search space grows exponentially with the number of neural network elements that are quantized. Generally, a brute-force search is technically infeasible in terms of compute power, compute time, and the like.

[0018] Existing methods for determining the bitwidth allocation per element of a neural network for MPQ generally are not applicable in all contexts and have technical performance deficiencies. For example, so-called “automatic mixed precision” (AMP) algorithms generally only work after training a neural network model, and thus are referred to as post-training quantization (PTQ) algorithms. By nature, AMP algorithms cannot account for hardware constraints or cross-element quantization costs (e.g., the effect of quantizing layer z on layer j). Further, AMP algorithms do not scale well for larger neural networks or for finer granularity.

[0019] As another example, “Bayesian bits” algorithms generally only work at training time, and thus are referred to as quantization-aware training (QAT) algorithms. Bayesian bits algorithms are generally unstable for PTQ. Further, Bayesian bits algorithms are generally limited to power-of-two bitwidths, thus limiting the available bitwidths for a particular quantizable element of a neural network.

[0020] Aspects described herein for performing MPQ using model sensitivity and constrained optimization overcome the technical problems of existing methods by determining near-optimal bitwidth allocations for neural network quantizers. In various aspects, this is accomplished by measuring the sensitivity of each element (e.g., layer) of a neural network to quantization and assigning quantization rates such that (1) resource constraints are satisfied, and (2) the total network quantization sensitivity is minimized. Examples of resource constraints include, for example, minimum bitwidth per layer and maximum binary operations (BOPS) or multiply-accumulate (MAC) operations across aneural network (or an elements thereof). Beneficially, the MPQ methods described herein are faster than conventional methods, scale linearly (rather than exponentially) with respect to the number of quantized elements of a neural network, and thus are amenable to both PTQ and QAT contexts. The beneficial technical effects of aspects described herein are reduced resource usage during QAT and PTQ as well as higher task performance for a given overall quantization rate compared to existing methods.Brief Introduction to Neural Network Quantization

[0021] In neural network quantization, the parameters and activations of a neural network may be quantized to a quantization grid. The quantization grid is, in general, not pre-defined, but rather part of the learning procedure for quantized neural networks. In aspects described herein, the following formulation for quantizer Q may be used:Q x; z, 8, b) = 8 x clamp ([- -]; w(b), Z(b) — z. (1)

[0022] In Equation 1, above, x denotes the quantizer input (network parameters and / or activations), z the zero-point of the quantizer, and 8 the step size. Further, u(b) and Z(b) map from the quantizer bitwidth to the upper and lower clamping threshold in the integer domain. These can differ depending on the quantizer specifics (e.g., (a)symmetric or (un)signed quantizers). Finally, [•] denotes a round-to-nearest integer mapping. Note that a quantizer may generally be associated with one or more nonoverlapping (e.g., unique) elements of a neural network model, such as one layer, a group of layers, a sub-layer, a group of sub-layers, a weight channel, a group of weight channels, and the like. A single element of a neural network, on the other hand, is generally associated with a single quantizer.

[0023] In aspects described herein, symmetric uniform quantization may be utilized. The step size 8 may be defined in terms of a range parameter <z, and a may either be inferred (e.g., in PTQ or at initialization) or learned (e.g., in QAT) using gradient based methods.

[0024] A common assumption is that the quantization error e = x — Q(x') is uniformly distributed with zero-mean and a standard deviation of 8 / f~ where 8 is the step size of the quantizer Q. This assumption can be shown to hold exactly under some conditions, but in practice Pseudo Quantization Noise (PQN) may be used and denoted as:

[0025] Following the PQN formulation, a PQN Quantizer QPmay be defined as follows:QP(X; z. S. b') = 8 ■ clamp

[0026] In Equation 2, e ~ [—1 / 2, 1 / 2] and everything else remains the same as in Equation 1. The use of the PQN formulation for QAT of neural networks has advantages in that the derivatives with respect to the bitwidth of the quantizer are non-zero. As such, the PQN formulation allows gradient-based optimization for the quantizer bitwidth (and, as a consequence, the step size).Aspects Related to Mixed Precision Neural Network Quantization

[0027] In MPQ, different elements of a network are quantized at different quantization rates, which allows for more efficient neural networks while maintaining task performance compared to standard homogeneously quantized neural networks. However, the MPQ bitwidth allocation search space grows exponentially in the number of quantizers (e.g., where one quantizer is associated with one element of a neural network to be quantized), which may make an exhaustive search computationally infeasible. For example, depending on configuration, ResNetl8 may employ approximately 50 quantizers (including weights and activations). Assuming four allowable bitwidths in this example, the search space is 450= 7.18E+23 combinations.

[0028] In MPQ, the optimization objective is to obtain a neural network that achieves optimal task performance (e.g. maximum classification accuracy or minimum perplexity) while minimizing resource requirements. This is a multi-objective optimization problem, and there is generally a trade-off between task performance and resource requirements.

[0029] Specifically, given a neural network JV', with parameters 0, that is quantized using K different quantizers Qlt... , Qk (e.g., one parameter and one activation quantizer per layer in this example), following an arbitrary order. Then, let the bitwidth allocation be given by b = [b^ b2, — , bK], where btis the bitwidth associated with quantizer Qt. The MPQ scalarized optimization objective £Q is then given as follows:£MPQ ®> IJ) = Z£(Q(0; ft)) + n(b) (3)where £(•) is an arbitrary task loss, zr (ft) denotes a resource cost, and > 0 is a scaling factor that weights the relative importance between the task loss and the resource cost. When optimizing Equation 3, the 0* and b* will depend on the curvature of £MPQ andnoguarantees can be given about the resource cost of the solution. Thus, a technical problem exists with existing methods.

[0030] Aspects described herein circumvent this technical problem by choosing a resource constraint and only optimizing for the task objective according to: minimize £.Q(Q, b') = L(Q(0, ft)) subjectto < 0

[0031] Note, for ease of presentation, this example is described with respect to parameter quantization only. However, the method generalizes to activation quantization as well. In Equation 4, ir(-) denotes the resource constraint. This encodes a similar resource cost as TT(-) in Equation 3. However, instead of minimizing the resource constraint, the aim is to satisfy only an upper bound constraint. This aligns with many real-world scenarios in which specific upper resource constraints must be satisfied.

[0032] Following the PQN model, the optimization objective may be approximated further as:£(Q(0, ft))

[0033] Here, 8 = [<5-^ ... , <5|0|] denotes the step size associated with each parameter. Following this formulation, the step size is a function of both the quantizer range a and the bitwidth ft, and the reparametrization in the second equality follows from this. Plugging Equation 5 back into Equation 4 and splitting the optimization over the network parameters and the bitwidth allocation into two separate minimizations, the following optimization objective is obtained:

[0034] This objective can directly be optimized using, for example, the penalty method or the augmented Lagrangian method. However, this is likely to require retraining of the underlying neural network model to convergence multiple times, which is costly. Instead, the approximation can be efficiently solved to the inner minimization problem,and the solution b* may be used to update 0 using common gradient-based QAT techniques.

[0035] Many methods for MPQ aim to solve optimization problems like Equations 3 and 6 directly using gradient-based methods. Although under many formulations, gradient-based methods are admitted, the bitwidth allocation is inherently a mixed integer optimization problem. For this reason, aspects described herein may solve an approximation of the inner bitwidth allocation minimization problem exactly. The inner optimization objective of Equation 6 is given as: b* = arg s. t. TT(ZJ) < 0. (7)

[0036] Taking the second-order Taylor expansion of Equation 7 with respect to 6 around zero gives the following approximation:

[0037] Here, the first-order term cancels out due to the expectation of multiplicative symmetric noise and the zeroth-order term is dropped since it does not depend on b. The obtained objective has no dependency on a pre-trained network having converged due to the assumption of pseudo quantization noise. Hence, this optimization problem can be solved during QAT — even when the outer optimization over 0 has not converged yet. Depending on the exact choice of resource constraints 7T, Equation 8 can be solved efficiently.

[0038] Putting everything together again, in some aspects the optimization procedure includes the following: first, solve Equation 8 to obtain b* , and second, perform a QAT gradient-based update step for 0 and a using:

[0039] Note that, although b* is a function of a, b* can be treated as constant considering the gradient updates for 0 and a.

[0040] Under some circumstances, Equation 8 can be solved greedily. For example, this may be the case, if the constraint is a total number of bits used in the bitwidth allocation. In many cases, however, this is not the case. Thus, solving Equation 8 as amixed integer program may be NP-HARD (nondeterministic polynomial time hard). However, practically speaking, a given neural network may generally have a reasonable number of quantizers (e.g., 50-40), which makes it feasible to solve a mixed integer program like Equation 8 for mixed precision neural network quantization. Further, it is possible to use efficient approximations. Depending on the resource constraints, this optimization problem can be efficiently solved using solvers, such as CVXOPT, GNU Linear Programming Kit (GLPK), Gurobi optimizer, MOSEK, Solving Constraint Integer Programs (SCIP), and others.

[0041] Note that when using a convex optimization framework, the constraints are required to be convex. Based on the exact optimization algorithms, the real requirements may be slightly less strict (e.g., interior point methods and convexity in the feasible set). While not all resource constraints will be convex, good convex approximations are generally available.Quantization Sensitivity in a Neural Network

[0042] In Equation 8, h is used as an approximation to the sensitivity to perturbations of activations or parameters. In particular, the diagonal of the Hessian of the neural network is used. In this specific optimization problem, the Hessian diagonal captures how sensitive a (group of) parameter(s) is to quantization, or perturbation(s) in general. A statistic that quantifies the how much the output of a neural network is affected by quantization may be referred to as a sensitivity. For example, if layer A of a network is more sensitive to quantization than layer B, the output of the full network changes more when quantizing layer A compared to quantizing layer B (or quantizing layer A at a higher rate relative to layer B). Thus, higher sensitivity means more sensitive to quantization, and a lower sensitivity the opposite.

[0043] Beneficially, aspects described herein may replace the Hessian in Equation 8 with any other sensitivity. There may be various reasons to do this, including: (1) computing the Hessian diagonal of a neural network is often expensive, and (2) the Hessian captures the effect of infinitesimal perturbations, whereas quantization perturbations may be larger. The first ratioanle may be addressed by using an approximation to the Hessian diagonal, called the Fisher Information Trace (FIT). However, other sensitivity metrics may also be used.

[0044] FIT is a sensitivity that has been shown to correlate well with the Hessian diagonal. For this reason, FIT may be used as a drop-in replacement for the more expensive Hessian diagonal. FIT is defined as: hL= [V0£(0)]2. (10)

[0045] Note that FIT, similar to the Hessian sensitivity, is computed based on the non-quantized neural network. As such, it requires an extra full precision forward and backward pass in order to compute and update sensitivities. Moreover, since gradients can only be obtained with respect to a subset of the training set, the computed FIT sensitivities are stochastic in nature. In order to reduce the computational requirements further and address the stochasticity in the sensitivity estimation, a running exponential average of FIT sensitivities may be used and update every T iterations.Minimizing Total Network Quantization Sensitivity

[0046] As above, one of the benefits of the methods described herein is that the total network quantization sensitivity is minimized. This objective can be formulated as follows. Let St(b) E [0, oo) be the sensitivity of quantizer Qtfor bitwidth b . Given quantizers Qj and Qk, Then, Sj b) > Sk(b') implies that setting Qj to bitwidth b harms the network more than setting Qkto bitwidth b'.

[0047] Now let # be the set of bitwidth candidates {B1, B2, ■■■ } (e.g., {2, 3. ..., 16}) and Q the set of quantizers. Given sensitivities St(Bj) for i = 1, ..., |Q| , and j = 1, the optimal bitwidth allocationis the solution for the following optimization problem:such that C(b1;*|Q|) < 0 (11)

[0048] One option for solving this optimization problem is to exhaustively search and evaluate the sensitivity to each quantized element of a neural network, such as in AMP methods. Another option is to perform Hessian aware quantization, which allows for the automatic selection of the relative quantization precision of each layer of a neural network, based on the layer’s Hessian spectrum. Another alternative is to obtain sensitivities from gradients, such as by using FIT, as described above.

[0049] Note that in aspects described herein, sensitivity estimation can be an independent process from quantization. As such, the choice of sensitivity estimation technique may be considered a modular aspect of the methods described herein.Choice of Resource Constraints and Optimization Methods

[0050] Different constraints can be used for mixed precision neural network quantization to obtain quantized neural networks for different real-world use cases. The following are some examples of constraints: total number of bits:network size in bits: x |T(| , where \7 | denotes tensor size; BOPS:MACSj x b^^ x bp^, where f(Z) and (Z) denote the input and parameter quantizer index for layer I and MACSj denotes the number of “multiply and accumulate” operations for layer I latency according to k(bi, ... , b\Q| ), where L is a differentiable latency model that takes parameter and activation bitwidths as input; peak memory usage; peak energy usage; total energy usage, total time; and others.Example Algorithm for Mixed Precision Neural Network Quantization

[0051] In some aspects, given neural networktask loss £(•), resource constraints ir(-), and bitwidth allocation update frequency T, the following algorithm may be used for mixed precision neural network quantization:Algorithm 1

[0052] In the above Algorithm 1, the bitwidth allocation update frequency parameterT allows for updating the bitwidth allocation less frequently than the neural network parameters are updated during training, as described above.Example Architecture for Performing Mixed Precision Neural Network Quantization

[0053] FIG. 1 depicts an example architecture 100 for performing mixed precision neural network quantization. Notably, FIG. 1 depicts an outer bound identifying aspects involved with QAT 102 as well as an inner bound identifying aspects involved with PTQ 104. It is evident that the single architecture beneficially works for both PTQ 102 and QAT 104, unlike existing methods (e.g., AMP and HAWQ).

[0054] During the PTQ 104, performing mixed precision neural network quantization begins at 106 with determining sensitivities (e.g.,with respect to Algorithm 1, above) for a set of quantizers associated with neural network 101, where each quantizer is associated with an element of the neural network 101, such as a layer of the neural network 101. As above, the sensitivities can be obtained via different methods, such as FIT, HAWQ, and AMP.

[0055] In one example, the sensitivities are determined according to gradients. For example, a full-precision forward pass of the neural network 101 is made followed by a full-precision backward pass. The backward pass provides gradients that then are used for a sensitivity-specific computation, such as FIT. For example, with FIT, the gradient is squared, but other transformations of the gradient could be used, such as the cube or log of the gradient in other examples.

[0056] Based on the sensitivities determined at 106, bitwidths can be allocated to the various aspects of the neural network 101 at 108; in other words, a bitwidth allocation for the neural network 101 is determined at 108.

[0057] In one example, the bitwidths are determined by solving an optimization problem with constraints, such as those formulated with reference to Equation 8 or Equation 11, above. In some cases, such as where a hardware constraint is included as one of the optimization constraints, a simulation may be run to determine a value of a constrained variable according to the bitwidth allocation.

[0058] In some cases, solving the optimization problem may be performed using fractional bitwidths (e.g., real value bitwidths, rather than integer value). This may be advantageous, especially in early training epochs, where the difference between adjacent integer bitwidths creates a significant change in model performance. Because whole integer bitwidths are ultimately used for the quantized neural network, the fractionalbitwidths can be converted to whole integers in several ways. For example, a fractional bitwidth can be rounded to the nearest integer. As another example, a final optimization step may be performed where the bitwidths are constrained to be integers. This final optimization step would generally follow a period in which fractional bits are used to enable smoother training.

[0059] In some aspects, determining the bitwidth allocation at 108 comprises selecting a bitwidth from a set of allowable bitwidths. The set of allowable bitwidths may include integer and / or fractional bitwidths in different implementations.

[0060] In some aspects, the set of allowable bitwidths is one of a plurality of sets of allowable bitwidths, and the set of allowable bitwidths is associated with an optimization or training iteration or epoch number. For example, it may be beneficial to limit the bitwidths possible during earlier iterations (e.g., to more than a threshold number of bits) to avoid over-quantizing elements of a neural network, and then to allow a larger set of bitwidths (e.g., including bitwidths less than the threshold) in later iterations. In some aspects, a bitwidth schedule may define allowable bitwidths according to ranges of training and / or optimization iterations, For example, for iterationsbitwidths bitwidth candidates {B1, B2, B3] are available, for iterations M ... N, bitwidth candidates {B1, B2, B3, B4, B5} are available, and for iterations > N , bitwidth candidates {B1(B2, B3, B4, B5, B6, B7} are available, etc.

[0061] Finally, the quantizations according to the bitwidths determined at 108 are applied to the neural network 101 at 110. Note that this process may be iterated any number of times, e.g., N number of times, or iterated until a stopping condition is met. For example, one stopping condition is when the calibration data is exhausted.

[0062] The QAT 102 adds an additional step 112 of evaluating the quantized neural network 101 and then updating parameters at 114 of the neural network 101 at 110.

[0063] In some cases, this is done by performing a forward pass of the quantized neural network 101 followed by a full-precision backwards pass of the neural network 101, if possible, or otherwise a backward pass using a gradient approximation to obtain gradients (sometimes referred to as “coarse gradients” when using a straight-through estimator). For example, a straight-through estimator (STE) may be used to approximate the gradients. Other examples are a REINFORCE algorithm or stochastic rounding. The gradients are then used to update parameters at 114 of the neural network 101.

[0064] Use of an STE during training of the quantized neural network 101 alleviates the problem of trying to minimize a piecewise constant function whose gradient is likely to vanish, which interferes with the standard back-propagation or chain rule. STEs provide an empirical way around this issue in the backward pass so that the gradient through the modified chain rule is non-trivial.

[0065] FIG. 2 depicts a mixed precision neural network quantization flow 200 as performed by, for example, the architecture 100 of FIG. 1.

[0066] The flow 200 begins at block 202 with a neural network in which mixed precision quantization is to be performed.

[0067] The flow 200 then proceeds to block 204 with performing a full-precision forward pass of the neural network. Here, full-precision refers to using unquantized parameters and / or activations for the neural network.

[0068] The flow 200 then proceeds to block 206 with performing a full-precision backward pass of the neural network to generate gradients.

[0069] The flow 200 then proceeds to block 208 with determining sensitivities for elements of the neural network, such as its layers, that are going to be quantized.

[0070] The flow 200 then proceeds to block 210 with solving an optimization problem of how to minimize resource usage by the neural network (e.g., total bits processed) while maximizing task performance of the neural network given constraints 220.

[0071] The flow 200then proceeds to block 212 with determining a bitwidth allocation according to the results of the optimization at block 210. The bitwidth allocation at block 212 may then be used to quantize the neural network.

[0072] Generally, blocks 202-212 depict an example of post training quantization (PTQ) as described above.

[0073] The flow 200 may then optionally proceed to block 214 with performing a forward pass of the quantized neural network.

[0074] The flow 200 may then optionally proceed to block 216 with performing a backward pass of the quantized neural network. As above, in some cases the backward pass may be performed with a gradient approximation method, such as STE or others.

[0075] The flow 200 may then optionally proceed to block 218 with determining gradients based on the backward pass at block 216. The gradients generated at block 218 may then be used to update the quantized neural network.

[0076] Generally, blocks 214-218 in conjunction with blocks 202-212 depict an example of quantization aware training of the neural network.Example Method for Performing Mixed Precision Quantization

[0077] FIG. 3 depicts an example method 300 of performing mixed precision quantization.

[0078] Method 300 begins at block 302 with determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more nonoverlapping elements of a machine learning model architecture.

[0079] The method 300 then proceeds to block 304 with determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints. For example, the optimization problem may be formulated as described above with respect to Equations 9 or 10.

[0080] The method 300 then proceeds to optional block 306 with quantizing the machine learning model architecture according to the determined bitwidth allocation for each of the one or more quantizers.

[0081] The method 300 then proceeds to optional block 308 with training the quantized machine learning model architecture. For example, block 308 may involve performing QAT as described above with respect to FIGS. 1 and 2.

[0082] In some aspects, the method 300 further includes performing an inference with the quantized machine learning model architecture.

[0083] In some aspects, the method 300 further includes performing a quantized precision forward pass of the machine learning model architecture; performing a backward pass of the machine learning model architecture (e.g., by STE or other gradient estimation technique) to generate a gradient for one or more parameters of the machine learning model architecture; and updating each respective parameter of the one or moreparameters of the machine learning model architecture based on a gradient associated with the respective parameter.

[0084] In some aspects, updating each respective parameter of the one or more parameters of the machine learning model architecture based on a gradient associated with the respective parameter is performed in each iteration of a number of iterations during a training epoch, and determining the sensitivity value for each of the one or more quantizers and determining a bitwidth allocation for each of the one or more quantizers iteratively is performed during only a subset of the number of iterations during the training epoch. For example, as described above, a bitwidth allocation update frequency parameter T allows for updating the bitwidth allocation less frequently than the neural network parameters are updated during training.

[0085] In some aspects, determining the sensitivity value for each of the one or more quantizers comprises: performing a full-precision forward pass of the machine learning model architecture; performing a full -preci si on backward pass of the machine learning model architecture to generate a gradient for each respective quantizer of the one or more quantizers; and determining an updated sensitivity value for each respective quantizer of the one or more quantizers based on the gradient associated with the respective quantizer.

[0086] In some aspects, updating the sensitivity value for each of one or more quantizers comprises performing a Hessian-based method, such as HAWQ.

[0087] In some aspects, updating the sensitivity value for each of one or more quantizers comprises performing an exhaustive evaluation method, such as AMP.

[0088] In some aspects, the one or more constraints comprise one or more of: a total number of quantized bits associated with the machine learning model architecture when quantized according to the bitwidth allocation; a total number of bits associated with the machine learning model architecture when quantized according to the bitwidth allocation and based on an input tensor size; a total number of binary operations (BOPS) associated with processing an input tensor through the machine learning model architecture; a latency of processing an input tensor through the machine learning model architecture; a peak memory usage of processing an input tensor through the machine learning model architecture; or an energy use of processing an input tensor through the machine learning model architecture.

[0089] In some aspects, the one or more quantizers comprise: at least one quantizer associated with a weight of the machine learning model architecture; and at least one quantizer associated with an activation of the machine learning model architecture. Further, in some aspects, the one or more quantizers comprise at least one quantizer associated with a weight channel.

[0090] In some aspects, the machine learning model architecture comprises a neural network model, such as a convolutional neural network model, a fully connected neural network model, and other types of neural network models.

[0091] In some aspects, the optimization problem is a convex optimization problem. In some aspects, solving the optimization problem includes relaxing constraints of a non- convex optimization problem to make it a convex optimization problem.

[0092] In some aspects, determining a bitwidth allocation for each of the one or more quantizers further comprises: determining a fractional bitwidth for one or more of the one or more quantizers; and determining an integer bitwidth for each fractional bitwidth.

[0093] In some aspects, the one or more non-overlapping elements of the machine learning model architecture comprises at least one of: one or more layers of the machine learning model architecture; one or more sub-layers of the machine learning model architecture; or one or more weight channels of the machine learning model architecture.

[0094] In some aspects, determining a bitwidth allocation for each of the one or more quantizers comprises selecting a bitwidth from a set of allowable bitwidths.

[0095] In some aspects, the set of allowable bitwidths is one of a plurality of sets of allowable bitwidths, and the set of allowable bitwidths is associated with a training iteration number.

[0096] In some aspects, the method 300, or any aspect related to it, may be performed by an apparatus, such as processing system 400 of FIG. 4, which includes various components operable, configured, or adapted to perform the method 300.

[0097] Note that FIG. 3 is just one example of a method, and other methods including fewer, additional, or alternative steps are possible consistent with this disclosure.Example Processing System for Performing Mixed Precision Quantization

[0098] FIG. 4 depicts aspects of an example processing system 400, which may be configured for performing mixed precision quantization, such as described herein with respect to FIGS. 1-3.

[0099] The processing system 400 includes a central processing unit (CPU) 402, which in some examples may be a multi-core CPU. Instructions executed at the CPU 402 may be loaded, for example, from a program memory associated with the CPU 402 or may be loaded from a memory partition 424.

[0100] In the depicted example, processing system 400 may further include one or more additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and a wireless connectivity component 412. In some examples, one or more of the processors of processing system 400 may be based on an ARM or RISC-V instruction set.

[0101] An NPU, such as 408, is generally a specialized circuit configured for implementing all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), and the like. An NPU may sometimes alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graph processing unit.

[0102] NPUs, such as 408, are generally configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models, as well as training of machine learning models. In some examples, a plurality of NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples they may be part of a dedicated neural -network accelerator.

[0103] NPUs may be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs optimized for performing both training and inference, the two tasks may still generally be performed independently.

[0104] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly compute-intensive operation thatinvolves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters (e.g., through backpropagation), such as weights and biases, in order to improve model performance. In supervised learning, an incorrect output from a machine learning model during training is backpropagated back through the machine learning model to determine gradients that are used to adjust model parameters to improve model task performance. In some cases, an NPU may be configured to perform the mixed precision quantization methods described herein.

[0105] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs may thus be configured to input a new piece of data and rapidly process it through an already trained model to generate a model output (e.g., an inference).

[0106] In some implementations, an NPU may be implemented as a part of one or more of CPU 402, GPU 404, and / or DSP 406.

[0107] In the depicted example, the processing system 400 further includes wireless connectivity component 412 for providing cellular, Wi-Fi, Bluetooth, and other wireless data transmission standards. The wireless connectivity component 412 is connected to one or more antennas 414.

[0108] In the depicted example, the processing system 400 further includes one or more sensor processing units 416 associated with any manner of sensor, one or more image signal processors (ISPs) 418 associated with any manner of image sensor, and / or a navigation processor 420, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0109] In the depicted example, the processing system 400 further includes one or more input and / or output devices 422, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, and the like.

[0110] In the depicted example, the processing system 400 further includes a computer-readable medium / memory 424, which is representative of one or more static and / or dynamic memories, such as a dynamic random access memory, a flash-based static memory, and the like. In this example, memory 424 includes computer-executable components, which may be executed by one or more of the aforementioned processors of processing system 400 for performing methods, such as those described herein, including those described with respect to FIGS. 1-3.[OHl] In particular, in this example, memory 424 includes one or more of sensitivity determination component 424A, bitwidth allocation component 424B, forward pass component 424C, backward pass component 424D, model training component 424E, model updating component 424F, model parameters 424G, quantizers 424H, constraints 4241, and optimization component 424J. The depicted components in memory 424, and others not depicted, may be configured to perform various aspects of the methods described herein.

[0112] Generally, the processing system 400 (or components thereof) may be configured to perform the methods described herein. For example, the sensitivity determination component 424A may be configured to determine the sensitivities (e.g., sensitivities 106 in FIG. 1) as in block 208 of FIG. 2 and block 302 of FIG. 3.

[0113] Further, the bitwidth allocation component 424B may be configured to allocate bitwidths (e.g., 110 in FIG. 1) to different elements of a machine learning model (e.g., neural network model 101 in FIG. 1 in FIG. 2), such as quantizers 424H, as in block 212 of FIG. 2 and block 304 of FIG. 3.

[0114] Further, the forward pass component 424C may be configured to perform a full or reduced precision forward pass through a machine learning model as in block 204 and 214 of FIG. 2 and as described above with respect to the method 300 of FIG. 3.

[0115] Further, the backward pass component 424D may be configured to perform a full precision or straight-through estimated backward pass through a machine learning model, such as a neural network, as in blocks 206 and 216 of FIG. 2 and as described above with respect to the method 300 of FIG. 3.

[0116] Further, the model training component 424E may be configured to perform training of a model, such as quantization aware training, as described above with respect to FIGS. 1-3

[0117] Further, the model updating component 424F may be configured to update parameters of a model (e.g., model parameters 114 in FIG. 1 and 424G), such as weights and biases, based on gradients (e.g., 218 in FIG. 2) as described above with respect to the method 300 of FIG. 3.

[0118] Further, the optimization component 424J may be configured to perform optimization based on constraints (e.g., 220 in FIG. 2 and 4241) in order to determineoptimal bitwidth allocations (e.g., 110 in FIG. 1 and 212 in FIG. 2), as described above with respect to block 304 of FIG. 3.

[0119] Notably, in other examples, aspects of the processing system 400 may be omitted or added. For example, the multimedia component 410, wireless connectivity 412, antennas 414, sensors 416, ISPs 418, and / or navigation component 420 may be omitted in other examples. Further, aspects of the processing system 400 maybe distributed between multiple devices in other examples.Example Clauses

[0120] Implementation examples are described in the following numbered clauses:

[0121] Clause 1 : A processor-implemented method of performing mixed precision quantization, comprising: determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

[0122] Clause 2: The processor-implemented method of clause 1, further comprising: performing a quantized precision forward pass of the machine learning model architecture; performing a backward pass of the machine learning model architecture to generate a gradient for one or more parameters of the machine learning model architecture; and updating each respective parameter of the one or more parameters of the machine learning model architecture based on a gradient associated with the respective parameter.

[0123] Clause 3: The processor-implemented method of clause 2, wherein: updating each respective parameter of the one or more parameters of the machine learning model architecture based on a gradient associated with the respective parameter is performed in each iteration of a number of iterations during a training epoch, and determining the sensitivity value for each of the one or more quantizers and determining a bitwidth allocation for each of the one or more quantizers iteratively is performed during only a subset of the number of iterations during the training epoch.

[0124] Clause 4: The processor-implemented method of any one of clauses 1-3, wherein determining the sensitivity value for each of the one or more quantizers comprises: performing a full-precision forward pass of the machine learning model architecture; performing a full-precision backward pass of the machine learning model architecture to generate a gradient for each respective quantizer of the one or more quantizers; and determining an updated sensitivity value for each respective quantizer of the one or more quantizers based on the gradient associated with the respective quantizer.

[0125] Clause 5: The processor-implemented method of any one of clauses 1-3, wherein updating the sensitivity value for each of one or more quantizers comprises performing a Hessian-based method.

[0126] Clause 6: The processor-implemented method of any one of clauses 1-3, wherein updating the sensitivity value for each of one or more quantizers comprises performing an exhaustive evaluation method.

[0127] Clause 7: The processor-implemented method of any one of clauses 1-6, wherein the one or more constraints comprise one or more of: a total number of quantized bits associated with the machine learning model architecture when quantized according to the bitwidth allocation; a total number of bits associated with the machine learning model architecture when quantized according to the bitwidth allocation and based on an input tensor size; a total number of binary operations (BOPS) associated with processing an input tensor through the machine learning model architecture; a latency of processing an input tensor through the machine learning model architecture; a peak memory usage of processing an input tensor through the machine learning model architecture; or an energy use of processing an input tensor through the machine learning model architecture.

[0128] Clause 8: The processor-implemented method of any one of clauses 1-7, wherein the one or more quantizers comprise: at least one quantizer associated with a weight of the machine learning model architecture; and at least one quantizer associated with an activation of the machine learning model architecture.

[0129] Clause 9: The processor-implemented method of clause 8, wherein the one or more quantizers comprise at least one quantizer associated with a weight channel.

[0130] Clause 10: The processor-implemented method of any one of clauses 1-9, wherein the machine learning model architecture comprises a neural network model.

[0131] Clause 11 : The processor-implemented method of clause 10, wherein the neural network model comprise a convolutional neural network model.

[0132] Clause 12: The processor-implemented method of any one of clauses 1-11, wherein the optimization problem is a convex optimization problem.

[0133] Clause 13: The processor-implemented method any one of clauses 1-12, wherein determining a bitwidth allocation for each of the one or more quantizers further comprises: determining a fractional bitwidth for one or more of the one or more quantizers; and determining an integer bitwidth for each fractional bitwidth.

[0134] Clause 14: The processor-implemented method of any one of clauses 1-13, wherein the one or more non-overlapping elements of the machine learning model architecture comprises at least one of: one or more layers of the machine learning model architecture; one or more sub-layers of the machine learning model architecture; or one or more weight channels of the machine learning model architecture.

[0135] Clause 15: The processor-implemented method of any one of clauses 1-14, wherein determining a bitwidth allocation for each of the one or more quantizers comprises selecting a bitwidth from a set of allowable bitwidths.

[0136] Clause 16: The processor-implemented method of clause 15, wherein: the set of allowable bitwidths is one of a plurality of sets of allowable bitwidths, and the set of allowable bitwidths is associated with a training iteration number.

[0137] Clause 17: The processor-implemented method of any one of clauses 1-16, further comprising quantizing the machine learning model architecture according to the determined bitwidth allocation for each of the one or more quantizers.

[0138] Clause 18: The processor-implemented method of clause 17, further comprising performing an inference with the quantized machine learning model architecture.

[0139] Clause 19: An apparatus, comprising: a memory comprising executable instructions; and a processor configured to execute the executable instructions and cause the apparatus to perform a method in accordance with any one of clauses 1-18.

[0140] Clause 20: An apparatus, comprising means for performing a method in accordance with any one of clauses 1-18.

[0141] Clause 21 : A non-transitory computer-readable medium storing program code configure for causing a processing system to perform a method according to any one of clauses 1-18.

[0142] Clause 22: A computer program product embodied on a computer-readable storage medium comprising code for performing a method in accordance with any one of clauses 1-18.Additional Considerations

[0143] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limiting of the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For instance, the methods described may be performed in an order different from that described, and various actions may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0144] The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combinationof computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, a system on a chip (SoC), or any other such configuration.

[0145] As used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

[0146] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.

[0147] As used herein, “coupled to” and “coupled with” generally encompass direct coupling and indirect coupling (e.g., including intermediary coupled aspects) unless stated otherwise. For example, stating that a processor is coupled to a memory allows for a direct coupling or a coupling via an intermediary aspect, such as a bus.

[0148] The methods disclosed herein comprise one or more actions for achieving the methods. The method actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims. Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor.

[0149] The following claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims. Within a claim, reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No claim element is to be construedunder the provisions of 35 U.S.C. §112(f) unless the element is expressly recited using the phrase “means for”. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.

Claims

WHAT IS CLAIMED IS:

1. A processor-implemented method of performing mixed precision quantization, comprising: determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

2. The processor-implemented method of claim 1, further comprising: performing a quantized precision forward pass of the machine learning model architecture; performing a backward pass of the machine learning model architecture to generate a gradient for one or more parameters of the machine learning model architecture; and updating each respective parameter of the one or more parameters of the machine learning model architecture based on a gradient associated with the respective parameter.

3. The processor-implemented method of claim 2, wherein: updating each respective parameter of the one or more parameters of the machine learning model architecture based on a gradient associated with the respective parameter is performed in each iteration of a number of iterations during a training epoch, and determining the sensitivity value for each of the one or more quantizers and determining a bitwidth allocation for each of the one or more quantizers iteratively is performed during only a subset of the number of iterations during the training epoch.

4. The processor-implemented method of one of claim 1, wherein determining the sensitivity value for each of the one or more quantizers comprises:performing a full-precision forward pass of the machine learning model architecture; performing a full-precision backward pass of the machine learning model architecture to generate a gradient for each respective quantizer of the one or more quantizers; and determining an updated sensitivity value for each respective quantizer of the one or more quantizers based on the gradient associated with the respective quantizer.

5. The processor-implemented method of any one of claim 1, wherein updating the sensitivity value for each of the one or more quantizers comprises performing a Hessian-based method.

6. The processor-implemented method of any one of claim 1, wherein updating the sensitivity value for each of the one or more quantizers comprises performing an exhaustive evaluation method.

7. The processor-implemented method of any one of claim 1, wherein the one or more constraints comprise one or more of: a total number of quantized bits associated with the machine learning model architecture when quantized according to the bitwidth allocation; a total number of bits associated with the machine learning model architecture when quantized according to the bitwidth allocation and based on an input tensor size; a total number of binary operations (BOPS) associated with processing an input tensor through the machine learning model architecture; a latency of processing an input tensor through the machine learning model architecture; a peak memory usage of processing an input tensor through the machine learning model architecture; or an energy use of processing an input tensor through the machine learning model architecture.

8. The processor-implemented method of any one of claim 1, wherein the one or more quantizers comprise: at least one quantizer associated with a weight of the machine learning model architecture; and at least one quantizer associated with an activation of the machine learning model architecture.

9. The processor-implemented method of claim 8, wherein the one or more quantizers comprise at least one quantizer associated with a weight channel.

10. The processor-implemented method of any one of claim 1, wherein the machine learning model architecture comprises a neural network model.

11. The processor-implemented method of claim 10, wherein the neural network model comprise a convolutional neural network model.

12. The processor-implemented method of any one of claim 1, wherein the optimization problem is a convex optimization problem.

13. The processor-implemented method of any one of claim 1, wherein determining a bitwidth allocation for each of the one or more quantizers further comprises: determining a fractional bitwidth for one or more of the one or more quantizers; and determining an integer bitwidth for each fractional bitwidth.

14. The processor-implemented method of any one of claim 1, wherein the one or more non-overlapping elements of the machine learning model architecture comprise at least one of: one or more layers of the machine learning model architecture; one or more sub-layers of the machine learning model architecture; or one or more weight channels of the machine learning model architecture.

15. The processor-implemented method of any one of claim 1, wherein determining a bitwidth allocation for each of the one or more quantizers comprises selecting a bitwidth from a set of allowable bitwidths.

16. The processor-implemented method of claim 15, wherein: the set of allowable bitwidths is one of a plurality of sets of allowable bitwidths, and the set of allowable bitwidths is associated with a training iteration number.

17. The processor-implemented method of claim 1, further comprising quantizing the machine learning model architecture according to the determined bitwidth allocation for each of the one or more quantizers.

18. The processor-implemented method of claim 17, further comprising performing an inference with the quantized machine learning model architecture.

19. A processing system, comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to: determine a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determine a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

20. The processing system of claim 19, wherein the one or more processors are further configured to execute the processor-executable instructions and cause the processing system to: perform a quantized precision forward pass of the machine learning model architecture; perform a backward pass of the machine learning model architecture to generate a gradient for one or more parameters of the machine learning model architecture; and update each of the one or more parameters of the machine learning model architecture based on its associated gradient.

21. The processing system of claim 20, wherein the one or more processors are further configured to execute the processor-executable instructions and cause the processing system to: update each of the one or more parameters of the machine learning model architecture based on its associated gradient in each iteration of a number of iterations during a training epoch, and determine the sensitivity value for each of the one or more quantizers and determine a bitwidth allocation for each of the one or more quantizers iteratively during only a subset of the number of iterations during the training epoch.

22. The processing system of any one of claim 19, wherein in order to determine the sensitivity value for each of the one or more quantizers, the one or more processors are further configured to execute the processor-executable instructions and cause the processing system to: perform a full-precision forward pass of the machine learning model architecture; perform a full-precision backward pass of the machine learning model architecture to generate a gradient for each respective quantizer of the one or more quantizers; and determine an updated sensitivity value for each respective quantizer of the one or more quantizers based on the gradient associated with the respective quantizer.

23. The processing system of any one of claim 19, wherein in order to update the sensitivity value for each of the one or more quantizers, the one or more processors are further configured to execute the processor-executable instructions and cause the processing system to performing a Hessian-based method.

24. The processing system of any one of claim 19, wherein in order to update the sensitivity value for each of the one or more quantizers, the one or more processors are further configured to execute the processor-executable instructions and cause the processing system to perform an exhaustive evaluation method.

25. The processing system of any one of claim 19, wherein the one or more constraints comprise one or more of:a total number of quantized bits associated with the machine learning model architecture when quantized according to the bitwidth allocation; a total number of bits associated with the machine learning model architecture when quantized according to the bitwidth allocation and based on an input tensor size; a total number of binary operations (BOPS) associated with processing an input tensor through the machine learning model architecture; a latency of processing an input tensor through the machine learning model architecture; a peak memory usage of processing an input tensor through the machine learning model architecture; or an energy use of processing an input tensor through the machine learning model architecture.

26. The processing system of any one of claim 19, wherein the one or more quantizers comprise: at least one quantizer associated with a weight of the machine learning model architecture; and at least one quantizer associated with an activation of the machine learning model architecture.

27. The processing system of claim 26, wherein the one or more quantizers comprise at least one quantizer associated with a weight channel.

28. The processing system of any one of claim 19, wherein the machine learning model architecture comprises a neural network model.

29. A non-transitory computer-readable medium comprising computerexecutable instructions that, when executed by a processor of a processing system, cause the processing system to perform a method comprising: determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more non-overlapping elements of a machine learning model architecture; and determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least:an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

30. A processing apparatus, comprising for: means for determining a sensitivity value for each of one or more quantizers, wherein each quantizer is associated with one or more nonoverlapping elements of a machine learning model architecture; and means for determining a bitwidth allocation for each of the one or more quantizers by solving an optimization problem defined by at least: an optimization objective of minimizing total sensitivity for the machine learning model architecture based on the bitwidth allocation; and one or more constraints.

Images