Hybrid precision quantization of machine learning model parameters

Abstract

Techniques and apparatus for improving machine learning model quantification are disclosed. A parameter tensor of a machine learning model is accessed, and a set of rows in the parameter tensor each including one or more outliers is identified. The parametric tensor is decomposed into a first parametric sub-tensor corresponding to the set of rows and a second parametric sub-tensor corresponding to at least one remaining row of the parametric tensor. The first parametric sub-tensor is quantized according to a first quantization scheme, and the second parametric sub-tensor is quantized according to a second quantization scheme. A quantized version of the machine learning model that includes the quantized first parametric sub-tensor and the quantized second parametric sub-tensor is generated.

Description

[0001] introduction

[0002] This disclosure relates to various aspects of machine learning.

[0003] A wide variety of machine learning model architectures have proliferated and have been used to provide solutions to a wide range of prediction problems. While specific architectures may vary, machine learning models generally rely on a set of model parameters with values ​​learned or trained based on training data, which can include labeled and / or unlabeled data. In many architectures (e.g., deep learning models), a large number of such parameters (in some cases, well over billions) are used to provide better utility. Additionally, in many cases, larger models (e.g., models with more parameters) tend to perform better (e.g., have higher prediction accuracy) and / or tend to be better suited to more complex prediction tasks. However, even relatively small models typically have a relatively large number of parameters and a considerable memory footprint.

[0004] Such a large number of parameters inherently leads to significant memory and / or storage device footprint, as well as similarly large usage of other computing resources. Model size becomes particularly problematic in resource-constrained scenarios where it is desirable to deploy trained models on devices with relatively limited resources, such as mobile devices, embedded devices, intelligent transportation vehicles, etc. Some conventional approaches to improve this problem involve parameter quantization. However, parameter quantization is an approximation-based process that inherently introduces errors into the model. Summary of the Invention

[0005] One aspect provides a method comprising: accessing a parameter tensor of a machine learning model; identifying rows in the parameter tensor, each including one or more outliers; decomposing the parameter tensor into a first parameter sub-tensor corresponding to the row sets and a second parameter sub-tensor corresponding to at least one remaining row in the parameter tensor; quantizing the first parameter sub-tensor according to a first quantization scheme; quantizing the second parameter sub-tensor according to a second quantization scheme; and generating a quantized version of the machine learning model including the quantized first parameter sub-tensor and the quantized second parameter sub-tensor.

[0006] One aspect provides a method comprising: accessing an input tensor of a layer of a machine learning model; decomposing the input tensor into a first input sub-tensor corresponding to a set of outlier indices and a second input sub-tensor corresponding to at least one remaining element in the input tensor; generating a first output sub-tensor based on multiplying the first input sub-tensor by a first parameter sub-tensor; generating a second output sub-tensor based on multiplying the second input sub-tensor by a second parameter sub-tensor; and generating an output tensor of the layer of the machine learning model based on the first output sub-tensor and the second output sub-tensor.

[0007] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0008] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0009] The accompanying drawings depict certain features of one or more aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0010] Figure 1 Example workflows for quantizing parameters of machine learning models according to various aspects of this disclosure are illustrated.

[0011] Figure 2 Example workflows for generating inferences using quantized machine learning model parameters are illustrated according to various aspects of this disclosure.

[0012] Figure 3 This is a flowchart depicting example methods for generating quantized machine learning models according to various aspects of this disclosure.

[0013] Figure 4 This is a flowchart depicting an example method for generating output tensors based on quantized machine learning model parameters, according to various aspects of this disclosure.

[0014] Figure 5 This is a flowchart depicting example methods for parameter quantization according to various aspects of this disclosure.

[0015] Figure 6 This is a flowchart depicting example methods for generating output tensors according to various aspects of this disclosure.

[0016] Figure 7 Example processing systems configured to perform various aspects of this disclosure are described.

[0017] Figure 8 Example processing systems configured to perform various aspects of this disclosure are described.

[0018] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0019] This disclosure provides apparatus, methods, processing systems, and computer-readable media for quantizing parameters of machine learning models.

[0020] Quantization of machine learning model parameters is a lossy compression technique that typically involves mapping high-precision parameters (e.g., weights encoded in floating-point representation) to a relatively small set of values ​​that can be represented using fewer bits. For example, sixteen-bit weights (e.g., weights encoded or stored using sixteen bits per weight) can be quantized to four-bit or eight-bit representations (e.g., quantized weights encoded or stored using four or eight bits per weight respectively), thereby significantly reducing the model's memory footprint. One method for quantization involves using a scaling factor and zeros (also referred to in some implementations as bias and / or offset) to map the original values ​​to values ​​that can be accurately stored in a desired bit width (smaller than the original bit width).

[0021] In some respects, bit width is chosen as a hyperparameter of the quantization process. In some respects, bit width is selected or determined based on the values ​​of the parameters being quantized (e.g., values ​​with larger standard deviations are assigned higher quantization bit widths). In some respects, some quantization parameters (e.g., zeros and scaling factors) can be similarly determined or learned based on the raw values ​​of the parameters. For example, various quantization parameters can be selected based on the mean and standard deviation of the raw parameters to map the raw values ​​to a set of quantized values ​​with mean and standard deviations that fit within the desired quantization bit width. However, since quantization is inherently based on approximation, these methods naturally introduce errors.

[0022] Furthermore, model accuracy can be significantly reduced by quantization when parameters include extrema (e.g., outliers). For example, some conventional methods involve pruning or removing such outlier parameters before quantization. This enables small-bit-width quantization, but introduces additional error through the loss of these parameter values. Some conventional methods involve preserving outliers during quantization. However, this can introduce substantial quantization loss because the quantization parameters are chosen to include these outliers. To mitigate such unacceptable quantization losses, some conventional methods alternatively rely on higher-bit quantization, which increases the model's memory footprint.

[0023] Various aspects of this disclosure provide mixed-precision quantization that uses multiple quantization schemes to achieve efficient parameter quantization to minimize (or at least reduce) memory usage, while also accommodating outlier parameter values. In some aspects, a first quantization scheme selected to accurately quantize outliers can be used to quantize such outliers, while a second quantization scheme selected to reduce the quantization parameter size to a minimum (or at least a reduced) amount can be used to quantize non-outliers. For example, in some aspects, the parameter corresponding to the outlier can be quantized to a first quantization bit width, while the remaining parameters can be quantized to a second quantization bit width smaller than the first quantization bit width. Although some examples described herein discuss the use of two bit widths (a first bit width for outlier parameters and a second bit width for non-outlier values), in some aspects, more than two bit widths can be used (e.g., a first bit width for non-outlier parameters, a second bit width for extreme outlier parameters, and a third bit width (between the first and second bit widths) for moderate outliers that are not as extreme as the extreme outlier parameters).

[0024] Some aspects of this disclosure are used to quantize the parameters used in matrix multiplication operations. Matrix multiplication is a common operation in various machine learning model architectures, where an input tensor (e.g., an activation data matrix from a previous layer in the model) is multiplied by a parameter tensor (e.g., a weight matrix). For example, feedforward layers or operations are typically implemented using matrix multiplication. Similarly, transformers (which involve self-attention) typically use multiple matrix multiplication operations to generate attention outputs. In many conventional architectures, the computational cost (e.g., memory and / or processing resources) for performing inference is primarily caused by matrix multiplication operations. Therefore, in some aspects, the parameter tensors (e.g., weight matrices) used in matrix multiplication operations can be decomposed based on identified outlier parameter values, and each sub-tensor can be quantized using appropriate quantization parameters to balance the reduction in parameter size with the preservation of model accuracy.

[0025] In some respects, during inference, the input tensor of the operation (e.g., the input to matrix multiplication) can be similarly decomposed based on outliers identified in the parameter tensor, and each sub-tensor of the input can be multiplied by a corresponding quantized sub-tensor from the parameter tensor. The resulting intermediate outputs can then be aggregated or combined (e.g., using element-wise summation) to generate the output tensor of the matrix multiplication operation.

[0026] Example workflow for parameter quantization of machine learning models

[0027] Figure 1An example workflow 100 for quantizing parameters of a machine learning model is illustrated. In some aspects, workflow 100 is performed by a quantization system (e.g., a computational system that performs model quantization on a trained machine learning model). For example, the quantization system may be a component of a training system (e.g., a component of a computational system that trains a machine learning model), a component of an inference system (e.g., a component of a computational system that uses the trained machine learning model to perform inference), or a separate system independent of the training and inference systems.

[0028] The illustrated workflow 100 is implemented by an outlier component 110, a decomposition component 120, a quantization component 130, and a compilation component 140. Although depicted as discrete components for clarity of concept, in some respects, the operations of the depicted components (and other components not illustrated) can be combined or distributed across any number and type of components and systems. Furthermore, the depicted components can generally be implemented using hardware, software, or a combination of hardware and software.

[0029] In the illustrated example, parameter tensor 105 is accessed by outlier component 110, which generates outlier index 115. Parameter tensor 105 typically corresponds to a subset of the parameters of a machine learning model. That is, parameter tensor 105 may include one or more parameters having values ​​learned during the model's training process (e.g., based on supervised and / or unsupervised training). For example, in some aspects, parameter tensor 105 includes a weight matrix for matrix multiplication operations. In some aspects, parameter tensor 105 typically corresponds to the original or non-quantized set of parameters from a non-quantized machine learning model.

[0030] In the illustrated example, outlier index 115 indicates the index (in parameter tensor 105) corresponding to a parameter having an outlier. As used herein, an "outlier" parameter is a parameter having a value that satisfies one or more outlier criteria. Typically, the specific definition of an outlier criterion can vary depending on a particular implementation. For example, in some aspects, a parameter is an outlier if its value exceeds a defined threshold value (e.g., a defined threshold above or below zero). In other aspects, a parameter is an outlier if it falls outside a defined range relative to the mean of the values ​​in parameter tensor 105. For example, a given parameter might be classified as an outlier if its value deviates from the mean of the values ​​in parameter tensor 105 by more than two standard deviations. Generally, any suitable criterion can be used to define an outlier.

[0031] In some respects, the outlier index 115 explicitly indicates the index of each outlier (e.g., the row and column of the outlier). In some respects, the outlier index is a row index indicating a row with one or more outliers, and / or a column index indicating a column with one or more outliers. For example, if the second row of parameter tensor 105 includes one or more outliers, then the outlier index 115 may indicate that the row index "two" corresponds to one or more outliers. It is worth noting that one or more other values ​​in the same row (e.g., in the second row) may have non-outliers. For example, the second row may contain fifteen non-outliers and one outlier. In some respects, the outlier index 115 may indicate that the second row includes at least one outlier, regardless of the other values ​​(non-outliers) in that row.

[0032] In some respects, the outlier component 110 can similarly generate from the parameter tensor 105 a set of non-outlier indices indicating rows that do not contain outliers. In other respects, the non-outlier indices can be implicitly inferred because the non-outlier rows inherently correspond to all other parameters or rows not indicated in the outlier index 115. For example, suppose the parameter tensor 105 comprises eight rows of parameters, and the first, third, fourth, and eighth rows contain one or more outliers. In some respects, the outlier index 115 can indicate the first, third, fourth, and eighth rows. The system can thus infer that the remaining rows (the second, fifth, sixth, and seventh rows) do not contain any outliers.

[0033] As illustrated, outlier index 115 is then provided to compilation component 140, which will be discussed in more detail below. Outlier index 115 is further accessed by decomposition component 120. In the illustrated example, decomposition component 120 additionally accesses parameter tensor 105. Decomposition component 120 decomposes parameter tensor 105 into a set of parameter subtensors 125A and 125B (collectively referred to as parameter subtensor 125) based on outlier index 115. As used herein, "decomposing" a tensor (also referred to in some respects as splitting, slicing, or marking boundaries) means distributing the elements of a tensor across two or more subtensors. For example, in the illustrated aspect, parameter subtensor 125A may contain any outliers found in parameter tensor 105 (e.g., as indicated by outlier index 115), and parameter subtensor 125B may contain the remaining (non-outlier) elements from parameter tensor 105.

[0034] In some aspects, as discussed above, outlier index 115 indicates which row(s) in parameter tensor 105 contains at least one outlier. In some such aspects, decomposition component 120 slices parameter tensor 105 to separate the rows indicated by outlier index 115 (each of which contains at least one outlier) to form parameter subtensor 125A. Decomposition component 120 uses the remaining rows (which are not indicated by outlier index 115 and do not contain any outliers) to form parameter subtensor 125B. That is, in some aspects, parameter subtensor 125B is a parameter (e.g., weight) matrix that does not contain outliers, and parameter subtensor 125A is a parameter (e.g., weight) matrix that contains any outliers found in parameter tensor 105. As discussed above, parameter subtensor 125A may additionally include one or more non-outliers (e.g., if non-outlier elements are in the same row as outlier elements in parameter tensor 105).

[0035] Although the illustrated example depicts decomposition component 120 decomposing parametric tensor 105 into two parametric subtensors 125, in some aspects, any number of subtensors may be used. For example, in some aspects, outlier component 110 may generate multiple sets of indices, such as a set of outlier indices indicating values ​​exceeding a first threshold (e.g., values ​​with weights that are more than two standard deviations away from the mean) and a second set of outlier indices indicating values ​​exceeding a second (lower) threshold but not exceeding a first (higher) threshold (e.g., values ​​with weights that are one to two standard deviations away from the mean). In some such aspects, decomposition component 120 may generate three parametric subtensors 125: a first for extreme outliers (e.g., values ​​more than two standard deviations away from the mean), a second for moderate outliers (e.g., values ​​one to two standard deviations away from the mean), and a third for the remaining (non-outlier) values.

[0036] In the depicted workflow 100, each of the parameter subtensors 125 is accessed by a quantization component 130, which generates corresponding quantized parameter subtensors 135A and 135B (collectively referred to as quantized parameter subtensors 135). Typically, the quantization component 130 can use any number and type of quantization scheme to quantize the parameter subtensors 125. In some aspects, the quantization component 130 uses a different quantization scheme for each parameter subtensor 125. For example, the quantization component 130 may determine a corresponding set of quantization parameters for each given parameter subtensor. As an example, the quantization component 130 may evaluate the values ​​contained in a given parameter subtensor 125 to determine an appropriate quantization scaling factor and / or zero point for the given parameter subtensor. In this way, quantization of parameter subtensor 135A (which may correspond to parameter subtensor 125A) can be performed using a first scale and zero point specifically selected for the values ​​reflected in parameter subtensor 125A, and quantization of parameter subtensor 135B (which may correspond to parameter subtensor 125B) can similarly be performed using a second scale and zero point specifically selected for the values ​​reflected in parameter subtensor 125B. In other respects, the quantization component uses at least two different quantization schemes for more than two parameter subtensors.

[0037] In some respects, the quantization component 130 may also select or determine an appropriate bit width for each corresponding parametric subtensor 125. For example, based on a predefined configuration and / or based on dynamic evaluation of the individual parametric subtensor 125, the quantization component 130 may determine to use a relatively smaller bit width for quantizing parametric subtensor 125B (which does not include outliers) compared to the bit width used to quantize parametric subtensor 125A (which includes one or more outliers). As discussed above, using a relatively larger bit width to quantize these outlier subtensors can achieve improved accuracy, while using a relatively smaller bit width for non-outliers (which do not benefit from the increased accuracy of a larger bit width) can minimize (or at least reduce) the total size (e.g., number of bits) of the collective quantization parameters.

[0038] For example, in some aspects, the quantization hyperparameter may indicate that the parameter subtensor 125A (which contains outliers) should be quantized to a one-bit width (e.g., 16 bits), while the parameter subtensor 125B (which does not contain outliers) should be quantized to a second bit width (e.g., 8 bits). In some aspects, in addition to using such pre-configured bit widths or instead of using such pre-configured bit widths, the quantization component 130 may dynamically determine an appropriate bit width. For example, based on the range and / or standard deviation of the values ​​in a given parameter subtensor 125, the quantization component 130 may select a bit width to balance the precision and size of the resulting quantized parameter subtensor 135.

[0039] As discussed above, although two quantized parameter subtensors 135 (generated using two discrete quantization schemes) are depicted for conceptual clarity, in some respects, the quantization component 130 can use any number and type of quantization schemes to generate any number of quantized parameter subtensors 135. For example, as discussed above, if three parameter subtensors 125 are generated, the quantization component 130 can generate three corresponding quantized parameter subtensors 135 (e.g., using the largest bit width for the parameter subtensor 125 containing the most extreme outliers, using a medium bit width for the parameter subtensor 125 containing moderate outliers, and using the smallest bit width for the parameter subtensor 125 containing no outliers).

[0040] In the illustrated example, the quantized parameter subtensor 135 is subsequently accessed by the compilation component 140. The compilation component 140 further accesses the outlier index 115 (generated by the outlier component 110) and generates a quantized machine learning model 145. For example, the compilation component 140 may aggregate, combine, or compile the quantized parameter subtensor 135 and the outlier index 115 as a machine learning model, enabling other systems to use the model for inference. In some aspects, as discussed in more detail below, the outlier index 115 may be included in the quantized machine learning model 145 or otherwise provided to the inference system because the outlier index 115 is used to decompose the input tensor during inference. For example, the outlier index 115 may be included as a hyperparameter for a portion (e.g., layer or other operation) of the model corresponding to the original parameter tensor 105. In some aspects, in addition to the outlier index, the compilation component 140 may similarly include the quantized parameters of each quantized parameter subtensor 135 in the compiled quantized machine learning model 145.

[0041] Although not included in the illustrated example, in some respects, compilation component 140 may similarly compile the remaining parameters and / or hyperparameters of the machine learning model to generate a quantized machine learning model 145. For example, workflow 100 may be executed for any other matrix multiplication operation, and the resulting quantized parameter subtensor 135 and outlier index 115 for each such matrix multiplication operation may be compiled to form a single quantized machine learning model 145. In some respects, other parameters and / or hyperparameters from the machine learning model (e.g., parameters that do not correspond to matrix multiplication operations) may similarly be compiled to form the quantized machine learning model 145. In some respects, such other parameters may be appropriately quantized (e.g., using workflow 100, or using any other suitable technique).

[0042] In this way, the quantized machine learning model 145 can represent or correspond to the original machine learning model (based on its plotted parameter tensor 105), but with a significantly reduced size due to the reduced bit width used to encode the parameters. This reduces the model's memory footprint, as well as the bandwidth consumed in sending all or part of the model across communication links (e.g., via a network such as the Internet to deploy the model to the inference system, and via links for retrieving parameters from memory and sending the retrieved parameters to the processing unit during inference). This significantly improves the operation of the computing devices involved.

[0043] Example workflow for generating inference using quantized machine learning model parameters.

[0044] Figure 2 An example workflow 200 for generating inference using quantized machine learning model parameters is illustrated. In some aspects, workflow 200 is performed by an inference system (e.g., a computational system that uses a trained machine learning model to perform inference). For example, the inference system may be a separate system from the training system (e.g., a system that trains the machine learning model) and / or the quantization system (e.g., a system that performs model quantization on the trained machine learning model). In other aspects, the inference system may be implemented as a component of another system, such as a component of the training system and / or a component of the quantization system.

[0045] The illustrated workflow 200 is implemented by a set of decomposition components 210, multiplication components 220A and 220B (collectively referred to as multiplication components 220), a set of dequantization components 230A and 230B (collectively referred to as dequantization components 230), and an aggregation component 240. Although depicted as discrete components for clarity of concept, in some respects, the operations of the depicted components (and other components not illustrated) can be combined or distributed across any number and type of components and systems. Furthermore, the depicted components can generally be implemented using hardware, software, or a combination of hardware and software.

[0046] In the illustrated example, input tensor 205 is accessed by decomposition component 210, which also accesses a set of outlier indices (e.g., outlier index 115 as illustrated). Input tensor 205 typically corresponds to a set of values ​​used as input to a layer or operation in a machine learning model. For example, input tensor 205 may include activation data output by activation functions of previous layers, or input tensor 205 may include feature data used as input to the model (e.g., if the operation is a first layer). In some aspects, input tensor 205 is received as input to matrix multiplication operations (e.g., for feedforward operations and / or attention operations), as discussed above.

[0047] Outlier index 115 typically indicates a parametric tensor (e.g., Figure 1The outlier index 115 is the index of any outlier value in the parameter tensor 105, as discussed above. That is, the outlier index 115 may have already been generated during the quantization of the machine learning model. In some aspects, the outlier index 115 explicitly indicates the outlier (e.g., the row and column of each outlier in the original parameter tensor). In some aspects, the outlier index 115 indicates the row in the parameter tensor that has one or more outliers. In some aspects, as discussed above, the outlier index 115 is provided as metadata or a hyperparameter associated with matrix multiplication operations in the model.

[0048] In the illustrated example, decomposition component 210 slices or decomposes input tensor 205 based on outlier index 115 to generate a set of input subtensors 215A and 215B (collectively referred to as input subtensor 215). For example, if outlier index 115 indicates a column of the original parameter tensor that includes one or more outliers, decomposition component 210 may slice input tensor 205 such that the corresponding row of input tensor 205 is assigned to input subtensor 215A, while the remaining rows (which are not indicated in outlier index 115 and therefore correspond to columns of parameters that do not have outliers) are assigned to input subtensor 215B.

[0049] As another example, if outlier index 115 indicates a row in the original parameter tensor that contains one or more outliers, decomposition component 210 can slice input tensor 205 such that the corresponding column of input tensor 205 is assigned to input subtensor 215A, while the remaining columns (which are not indicated in outlier index 115 and therefore correspond to rows of parameters without outliers) are assigned to input subtensor 215B. For example, if outlier index 115 indicates that the first, third, and fourth rows of the parameter tensor contain outliers, decomposition component 210 can assign the first, third, and fourth columns of input tensor 205 to input subtensor 215A. In this way, by performing column-by-column slicing (while the parameter tensor is sliced ​​row-by-row), decomposition component 210 ensures that the width of input subtensor 215 matches the height of the corresponding quantized parameter subtensor, and that each value in input tensor 205 remains aligned with its corresponding parameter in the parameter tensor for matrix multiplication.

[0050] As discussed above, although two input subtensors 215 are depicted for clarity of concept, in all respects, the decomposition component 210 can slice the input tensor 205 into any number of subtensors (based on the outlier index 115). As illustrated, the input subtensors 215 are accessed by corresponding multiplication components 220. Specifically, input subtensor 215A is accessed by multiplication component 220A, and input subtensor 215B is accessed by multiplication component 220B. Although two multiplication components 220 (one for each input subtensor 215) are depicted for clarity of concept, in some respects, the inference system can use a single multiplication component 220 (e.g., processing the input subtensors 215 sequentially). In other respects, multiple multiplication components 220 can be used, wherein at least one of the multiplication components processes multiple input subtensors sequentially.

[0051] As illustrated, each multiplication component 220 further accesses the quantization parameter subtensor corresponding to the input subtensor 215 (e.g., quantization parameter subtensor 135 as shown in the figure). Specifically, multiplication component 220A accesses the quantization parameter subtensor 135A corresponding to the input subtensor 215A (e.g., because both quantization parameter subtensor 135A and input subtensor 215A correspond to outlier index 115). Furthermore, multiplication component 220B accesses the quantization parameter subtensor 135B corresponding to the input subtensor 215B (e.g., because both quantization parameter subtensor 135B and input subtensor 215B correspond to residual values ​​not indicated by outlier index 115).

[0052] In the illustrated workflow 200, each of the multiplication components 220 then performs matrix multiplication to multiply each input subtensor 215 with the corresponding quantization parameter subtensor 135. Specifically, multiplication component 220A multiplies input subtensor 215A with quantization parameter subtensor 135A to generate output subtensor 225A. Furthermore, multiplication component 220B multiplies input subtensor 215B with quantization parameter subtensor 135B to generate output subtensor 225B.

[0053] As illustrated, these output subtensors 225A and 225B (collectively referred to as output subtensors 225) are accessed by dequantization components 230A and 230B, respectively. Specifically, output subtensor 225A (which corresponds to or is generated based on outlier parameter values) is accessed by dequantization component 230A, and output subtensor 225B (which corresponds to non-outlier parameter values) is accessed by dequantization component 230B. Although two dequantization components 230 (one for each output subtensor 225) are depicted for conceptual clarity, in some respects, the inference system can use a single dequantization component 230 (e.g., processing the output subtensors 225 sequentially).

[0054] In the illustrated example, dequantization component 230 performs a dequantization operation on output subtensor 225 to generate dequantized output subtensors 235A and 235B (collectively referred to as dequantized output subtensor 235). Specifically, dequantization component 230A dequantizes output subtensor 225A based on the quantization parameters used to quantize the quantization parameter subtensor 135A to generate dequantized output subtensor 235A, and dequantization component 230B dequantizes output subtensor 225B based on the quantization parameters used to quantize the quantization parameter subtensor 135B to generate dequantized output subtensor 235B.

[0055] The dequantized output subtensor 235 is then accessed by the aggregation component 240, which generates the output tensor 245. In some aspects, the specific aggregation operation used by the aggregation component 240 may vary depending on the specific implementation. For example, the aggregation component 240 may use summation, concatenation, averaging, etc. In the illustrated example, the aggregation component 240 performs element-wise summation on the dequantized output subtensor 235 to generate the output tensor 245.

[0056] Output tensor 245 can be used as the output of matrix multiplication (or other operations) corresponding to the quantized parameter subtensor. For example, output tensor 245 can be provided as the output of a model, and / or as the output of subsequent layers or operations in a machine learning model (e.g., as part of a feedforward layer or attention component, as discussed above).

[0057] Using workflow 200, the inference system can perform efficient matrix multiplication using mixed-precision quantized parameters. As discussed above, this mixed-precision representation can represent or correspond to the original machine learning model (based on its plotted parameter tensor), with a significantly reduced size due to the reduced bit width used to encode the parameters. This reduces the model's memory footprint and the bandwidth consumed in sending all or part of the model across communication links (e.g., from memory to the processing unit used to perform multiplication during inference). This significantly improves the operation of the computing devices involved.

[0058] Example methods for generating quantized machine learning models

[0059] Figure 3 This is a flowchart depicting an example method 300 for generating quantized machine learning models. In some aspects, method 300 is derived from a quantization system (such as the one mentioned above). Figure 1 The discussion focuses on the quantitative system.

[0060] At box 305, the quantized system access parameter tensor (e.g., Figure 1The parameter tensor 105). As discussed above, parameter sensors typically include one or more parameters for one or more components or parts of a machine learning model. For example, a parameter tensor may correspond to a weight matrix used to perform matrix multiplication operations within the model (e.g., as part of an attention mechanism or operation, or as part of a feedforward layer of a neural network). Typically, any number of parameter tensors can exist in a given model, each with any number of parameters. In some aspects, the quantization system may perform method 300 to quantize the model for each such tensor (e.g., sequentially or in parallel). In some aspects, as discussed above, the parameters or elements in a parameter tensor may have values ​​learned or trained during the training process of the model (e.g., based on labeled and / or unlabeled training data).

[0061] At box 310, the quantization system identifies the outlier index of the parameter tensor (e.g., Figure 1 An outlier index is a set of indices (115). In some respects, as discussed above, an outlier index typically indicates the location of any parameter (in a tensor) that has an outlier. In some respects, outliers are defined relative to other values ​​in the parameter tensor. For example, a parameter may be designated as an outlier if its value deviates from the mean of the parameter tensor by more than two standard deviations. In other respects, outliers are defined using a fixed threshold (rather than relative to other values ​​in the tensor). For example, a parameter may be marked as an outlier if its value exceeds a static threshold (e.g., a value greater than ten). Typically, various criteria are used to define outliers in quantization systems.

[0062] In some respects, as discussed above, outlier indexes are row indexes, where each indicated row includes at least one outlier parameter value. That is, if the parameter tensor is a weight matrix, then an outlier index can indicate a row with at least one outlier value.

[0063] At box 315, the quantization system decomposes the parametric tensor based on outlier indices to generate parametric subtensors (e.g., ...). Figure 1 The parametric subtensor (125). For example, in some aspects, the quantization system creates a first parametric subtensor and a second parametric subtensor. The first parametric subtensor may contain the value indicated in the outlier index (e.g., by stacking or concatenating the indicated rows from the parametric tensor to generate a new matrix). The second parametric subtensor may contain residual values ​​(e.g., by stacking or concatenating rows from the parametric tensor that are not indicated in the outlier index). As discussed above, although non-outlier subtensors do not contain outliers, outlier subtensors may contain one or more non-outliers (in addition to all outliers from the parametric tensor). Although the decomposition of a parametric tensor into two subtensors is described for conceptual clarity, as discussed above, the quantization system may decompose a parametric tensor into any number of subtensors.

[0064] At box 320, the quantization system selects one of the parametric subtensors. Typically, the quantization system can use any suitable technique or criterion to select the subtensor, as it will process each subtensor during method 300. Additionally, although depicted as an iterative process for clarity (where the quantization system sequentially selects and processes each subtensor), in some respects, the quantization system can process some or all of the subtensors in parallel.

[0065] At box 325, the quantization system determines or selects a quantization scheme for the selected subtensor. In some aspects, as discussed above, the quantization system determines the quantization scheme by identifying a definition scheme corresponding to the selected subtensor (e.g., indicated in a previously generated or defined mapping). For example, the quantization system may use a first definition scheme for any non-outlier subtensor and a second definition scheme for any subtensor containing outliers. In some aspects, as discussed above, the quantization system may determine the quantization scheme based on the distribution of values ​​in the selected subtensor. For example, the quantization system may select or define a quantization scheme based on the mean, range of values, standard deviation of values, minimum and / or maximum values, etc. This allows the quantization system to dynamically quantize each subtensor using custom quantization for each subtensor.

[0066] In some respects, as discussed above, quantization schemes typically include a quantization scale (e.g., chosen based on the range of values ​​in the subtensor), a quantization zero or offset (e.g., chosen based on the average value in the subtensor), and a quantization bit width (e.g., chosen based on the range of values ​​in the subtensor and / or based on a mapping defined according to whether the subtensor contains outliers). For example, as discussed above, a quantization system may choose a quantization scheme with a relatively high bit width for subtensors containing outliers (e.g., using more bits to encode each quantization parameter) and a quantization scheme with a relatively low bit width for non-outlier subtensors (e.g., using fewer bits to encode each quantization parameter).

[0067] At box 330, the quantization system uses the determined quantization scheme to quantize the selected parametric subtensor (e.g., to generate...). Figure 1 The quantized parameter subtensor (135). As discussed above, a quantized subtensor typically involves encoding each parameter with a relatively small bit width compared to the unquantized parameters. For example, for each parameter, the quantization system can ( i Multiply the parameters by the determined quantization ratio and / or ( ii Add or subtract zeros. The resulting value can then be encoded using the determined bit width. In this way, a quantized subtensor can be represented using relatively fewer bits compared to a non-quantized subtensor.

[0068] At box 335, the quantization system determines whether there exists at least one additional parameter subtensor that has not yet been quantized (generated at box 315). If yes, method 300 returns to box 320 to select the parameter subtensor for processing. If no, method 300 continues to box 340.

[0069] At box 340, the quantization system generates a quantized machine learning model based on quantized subtensors. For example, as discussed above, the quantization system can compile quantized subtensors and associated metadata (such as quantization parameters for each subtensor, outlier indices, etc.) to represent the original parameter tensors in the quantized model. In some aspects, generating a quantized machine learning model similarly involves compiling quantized parameters (and other related data) of other parameters in the original model (e.g., other parameter tensors and / or other layers or parts of the model).

[0070] In this way, the quantization system can dynamically generate quantized versions of the (original) machine learning model that can be represented or stored using fewer bits. This reduces the model's memory footprint and the communication bandwidth consumed by transmitting the quantized model (e.g., via a network to the inference system, and / or from memory to the processing components of the inference system). In this manner, the quantized model can be used for inference with reduced computational resources, enabling more efficient execution and / or use of the model on resource-constrained devices.

[0071] Example methods for generating output tensors based on quantized machine learning model parameters.

[0072] Figure 4 This is a flowchart depicting an example method 400 for generating output tensors based on quantized machine learning model parameters. In some aspects, method 400 is powered by an inference system (such as the one mentioned above). Figure 2 The inference system discussed is executed.

[0073] At box 405, the inference system accesses the input tensor (e.g., input tensor 205). As discussed above, input tensors are typically used as inputs to layers or portions of a machine learning model (e.g., as inputs to matrix multiplication operations, such as in feedforward and / or attention layers). As discussed above, input tensors are typically tensors (e.g., vectors, matrices, or scalar values) that include one or more data elements, which are used as inputs to operations (e.g., matrix multiplication) of the machine learning model. For example, if the operation is the first layer of the model, the input tensor could be data provided as input to the model (e.g., sensor data). If the operation is part of an inner layer (e.g., an attention operation or feedforward layer within the model), the input tensor could include data output from previous layers or operations (e.g., activation data from previous layers).

[0074] At box 410, the inference system determines a set of outlier indices corresponding to the current operation or layer of the model. That is, if an input tensor is received or accessed for processing using a given layer or operation, the inference system may identify outlier indices generated for that given layer or operation. In some respects, as discussed above, these outlier indices may be determined during the quantization of the model (e.g., by the quantization system). These outlier indices may be provided or included as part of the compiled quantized model.

[0075] As discussed above, in some respects, outlier indices indicate rows in the original parameter tensor that contain one or more outlier parameter values. This allows inference systems to identify elements in the input tensor corresponding to outlier rows.

[0076] At box 415, the inference system decomposes the input tensor into input subtensors based on the outlier index (e.g., Figure 2 The set of input subtensors (215). For example, if the outlier index corresponds to a specific parameter (e.g., a specific column for each outlier). Hexing If an index is used, the inference system can use these indexes to identify the corresponding input elements (e.g., the same column). Hexing The input element at the location, or the reverse index column. Hexing (Input elements at the location). In some respects, if the outlier index is a row index (e.g., each index indicates a row from the original parameter tensor that includes at least one outlier), the inference system can identify a corresponding column for each such row.

[0077] For example, if the outlier index indicates a row Including one or more outlier values ​​in the parameter tensor, the inference system can generate columns that include those from the input tensor. The input tensor is the input subtensor. In some respects, in addition to the outlier subtensor (the input subtensor containing elements corresponding to the parameters in the outlier parameter subtensor), the inference system similarly generates non-outlier subtensors containing the remaining elements from the input tensor. Although the decomposition of the input tensor into two subtensors is described for conceptual clarity, as discussed above, the inference system can decompose the input tensor into any number of subtensors.

[0078] At box 420, the inference system selects one of the input subtensors. Typically, the inference system can use any suitable technique or criterion to select the subtensor, as it will process each subtensor during method 400. Additionally, although depicted as an iterative process for clarity (where the inference system selects and processes each subtensor sequentially), in some respects, the inference system can process some or all of the subtensors in parallel.

[0079] At box 425, the inference system accesses the parametric subtensor corresponding to the selected subtensor. For example, if the selected subtensor corresponds to an outlier index, the inference system can access the parametric subtensor that also corresponds to the outlier index. Similarly, if the selected subtensor contains the remaining elements from the input tensor (those that do not correspond to outlier indices), the inference system can access the parametric subtensor that also corresponds to the remaining (non-outlier) parameters. In some respects, as discussed above, the parametric subtensor can be a quantized subtensor (e.g., quantized by a quantization component based on a set of quantization parameters). That is, at box 425, the inference system can access the corresponding quantized parametric subtensor (e.g., Figure 1 (One of the quantization parameter subtensors 135).

[0080] At box 430, the inference system generates an output subtensor based on the selected input subtensor and its corresponding parameter subtensor. For example, in the case of matrix multiplication, the inference system can multiply the selected input subtensor with its corresponding parameter subtensor.

[0081] At box 435, the inference system dequantizes the generated output subtensor based on the quantization parameters used to quantize the parameter subtensor (accessed at box 425). In some respects, as discussed above, these quantization parameters may be provided or included as part of the quantization machine learning model, thereby allowing the inference device to properly dequantize the parameters.

[0082] At box 440, the inference system determines whether at least one additional input subtensor remains to be processed. If yes, method 400 returns to box 420. If no, method 400 continues to box 445.

[0083] At box 445, the inference system aggregates the output sub-tensors to generate the output tensor (such as...). Figure 2 The output tensor (245). For example, in some aspects, the inference system can compute element-wise summation of the output subtensors to generate the output tensor. This output tensor corresponds to the output of the current operation or layer in the machine learning model (e.g., the output of a matrix multiplication operation).

[0084] In this way, compared to at least some conventional machine learning models, the inference system can generate machine learning model outputs and / or internal values ​​(e.g., the results of matrix multiplication operations) with significantly reduced computational overhead. Although not included in the illustrated example, in some respects, the inference system can similarly perform method 400 for each other relevant operation or layer in the quantization model (e.g., for each other matrix multiplication operation).

[0085] Example methods for parameter quantization

[0086] Figure 5 This is a flowchart depicting an example method 500 for parameter quantization. In some aspects, method 500 consists of a quantization system (such as the one mentioned above). Figure 1 and / or Figure 3 The discussion focuses on the quantitative system.

[0087] At box 505, access the parameter tensor of the machine learning model.

[0088] In some respects, parameter tensors include weights used for matrix multiplication operations.

[0089] In some respects, matrix multiplication is performed as part of at least one of the following: a feedforward operation of a machine learning model, or an attention operation of a machine learning model.

[0090] At box 510, identify the set of rows in the parameter tensor that each contains one or more outliers.

[0091] In some aspects, method 500 also includes identifying one or more outliers, including: determining one or more outlier criteria; and using the one or more outlier criteria to evaluate each corresponding value in the parameter tensor.

[0092] In some respects, one or more outlier criteria include a magnitude threshold.

[0093] In some aspects, method 500 also includes generating a set of outlier indexes comprising the corresponding index of each corresponding row in the row set. This set of outlier indexes can be used to decompose the input tensor during inference.

[0094] At box 515, the parameter tensor is decomposed into a first parameter subtensor corresponding to the row set and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor.

[0095] At box 520, the first parameter subtensor is quantized according to the first quantization scheme.

[0096] At box 525, the second parameter subtensor is quantized according to the second quantization scheme.

[0097] In some respects, the first quantization scheme includes a first quantization bit width, and the second quantization scheme includes a second quantization bit width that is smaller than the first quantization bit width.

[0098] At box 530, a quantized version of the machine learning model is generated, which includes a quantized first parameter subtensor and a quantized second parameter subtensor.

[0099] In some aspects, the inference includes: accessing an input tensor; decomposing the input tensor into a first input sub-tensor having columns corresponding to a set of outlier indexes and a second input sub-tensor corresponding to at least one remaining column in the input tensor; generating a first output sub-tensor based on multiplying the first input sub-tensor with a first parameter sub-tensor; and generating a second output sub-tensor based on multiplying the second input sub-tensor with a second parameter sub-tensor.

[0100] In some aspects, the inference also includes: dequantizing the first output subtensor and the second output subtensor; and generating the output tensor by element-wise summing the first dequantized output subtensor and the second dequantized output subtensor.

[0101] Example generation of output tensors

[0102] Figure 6 This is a flowchart depicting an example method 600 for generating output tensors. In some aspects, method 600 is powered by an inference system (such as the one mentioned above). Figure 2 and / or Figure 4 The inference system discussed is executed.

[0103] At box 605, access the input tensor of the layer of the machine learning model.

[0104] At box 610, the input tensor is decomposed into a first input subtensor corresponding to the set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor.

[0105] In some respects, the outlier index set indicates the set of rows in the non-quantized parameter tensors of the layers of a machine learning model that each contains one or more outliers.

[0106] At box 615, the first output subtensor is generated by multiplying the first input subtensor with the first parameter subtensor.

[0107] At box 620, the second output subtensor is generated by multiplying the second input subtensor with the second parameter subtensor.

[0108] In some respects, the first parameter subtensor is quantized according to a first quantization scheme, and the second parameter subtensor is quantized according to a second quantization scheme.

[0109] In some respects, the first quantization scheme includes a first quantization bit width, and the second quantization scheme includes a second quantization bit width that is greater than the first quantization bit width.

[0110] In some respects, the first-parameter subtensor and the second-parameter subtensor include weights used for matrix multiplication operations.

[0111] In some respects, matrix multiplication is performed as part of at least one of the following: a feedforward operation of a machine learning model, or an attention operation of a machine learning model.

[0112] At box 625, the output tensors of the machine learning model's layers are generated based on the first and second output sub-tensors.

[0113] In some respects, generating the output tensor includes: dequantizing the first output subtensor and the second output subtensor; and summing the first dequantized output subtensor and the second dequantized output subtensor element by element.

[0114] Example processing system for data quantization

[0115] In some respects, refer to Figures 1 to 6 The described workflows, techniques, and methods can be implemented on one or more devices or systems. Figure 7 Various aspects configured to perform this disclosure are described (including, for example, references to...). Figures 1 to 6 The described techniques and methods are used in the example processing system 700. In some aspects, the processing system 700 may correspond to a quantization system (such as the one mentioned above). Figure 1 and / or Figure 3 (The quantization system discussed below). For example, processing system 700 may correspond to a system that quantizes a machine learning model after training. Although depicted as a single system for clarity of concept, in some respects, as discussed above, the operations described below with respect to processing system 700 can be distributed across any number of devices or systems.

[0116] The processing system 700 includes a central processing unit (CPU) 702, which in some examples may be a multi-core CPU. Instructions executed at the CPU 702 may be loaded, for example, from program memory associated with the CPU 702 or from a memory partition (e.g., a partition of memory 724).

[0117] The processing system 700 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, a multimedia component 710 (e.g., a multimedia processing unit), and a wireless connectivity component 712.

[0118] NPUs (such as the NPU 708) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0119] NPUs such as the NPU 708 are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0120] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0121] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0122] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0123] In some implementations, the NPU 708 is part of one or more of the CPU 702, GPU 704, and / or DSP 706.

[0124] In some examples, the wireless connectivity component 712 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and / or other wireless data transmission standards. The wireless connectivity component 712 is further coupled to one or more antennas 714.

[0125] The processing system 700 may also include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISPs) 718 associated with any type of image sensor, and / or a navigation processor 720, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0126] The processing system 700 may also include one or more input and / or output devices 722, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0127] In some examples, one or more processors in the processing system 700 may be based on the ARM or RISC-V instruction set.

[0128] The processing system 700 also includes a memory 724, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 724 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 700.

[0129] Specifically, in this example, memory 724 includes an outlier component 724A, a decomposition component 724B, a quantization component 724C, and a compilation component 724D. Memory 724 also includes components for one or more models (e.g., Figure 1 The model parameters 724E are a parameter tensor of 105. Although not included in the illustrated example, in some respects, memory 724 may also include other data, such as training data (e.g., for training and / or fine-tuning the model). Although for the sake of conceptual clarity, Figure 7 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) may be implemented together or separately.

[0130] The processing system 700 also includes an outlier circuit 726, a decomposition circuit 727, a quantization circuit 728, and a compiler circuit 729. The depicted circuits and other circuits not depicted can be configured to perform various aspects of the techniques described herein.

[0131] For example, outlier component 724A and / or outlier circuit 726 (which may correspond to...) Figure 1 Outlier component 110 can be used to identify outlier parameters in model parameter 724E and / or generate outlier indexes, as discussed above. For example, outlier component 724A and / or outlier circuit 726 can use various outlier criteria to identify outliers and generate outlier indexes indicating rows in one or more parameter tensors that contain such outliers.

[0132] Decomposition component 724B and / or decomposition circuit 727 (which may correspond to Figure 1 The decomposition component 120 can be used to decompose a parametric tensor based on outlier indices, as discussed above. For example, the decomposition component 724B and / or the decomposition circuit 727 can decompose a parametric tensor by generating a sub-tensor containing rows of the parametric tensor that include one or more outliers and generating a second sub-tensor containing the remaining rows of the parametric tensor.

[0133] Quantization component 724C and / or quantization circuit 728 (which may correspond to) Figure 1 The quantization component 130 can be used to quantize the parametric subtensors, as discussed above. For example, the quantization component 724C and / or the quantization circuit 728 can determine or select a quantization scheme for each parametric subtensor, thereby quantizing each corresponding subtensor according to the appropriate quantization scheme.

[0134] Compiler component 724D and / or compiler circuit 729 (which may correspond to) Figure 1 The compiler component 140 can be used to compile quantization parameter subtensors and related metadata, such as outlier indices and quantization parameters, as discussed above. For example, compiler component 724D and / or compiler circuit 729 can aggregate a set of quantization parameter subtensors and corresponding outlier indices and quantization parameters for each operation in a machine learning model to generate a quantized version of the machine learning model.

[0135] Although for clarity Figure 7 While depicted as separate components and circuits, the outlier circuit 726, decomposition circuit 727, quantization circuit 728, and compilation circuit 729 may be implemented jointly or separately in other processing devices of the processing system 700, such as within the CPU 702, GPU 704, DSP 706, NPU 708, etc.

[0136] Generally speaking, the processing system 700 and / or its components can be configured to perform the methods described herein.

[0137] It is worth noting that, in other aspects, such as when the processing system 700 is a server computer, components of the processing system 700 can be omitted. For example, in other aspects, the multimedia component 710, the wireless connectivity component 712, the sensor processing unit 716, the ISP 718, and / or the navigation processor 720 can be omitted. Furthermore, various aspects of the processing system 700 can be distributed among multiple devices.

[0138] Example processing system for inference

[0139] In some respects, refer to Figures 1 to 6The described workflows, techniques, and methods can be implemented on one or more devices or systems. Figure 8 Various aspects configured to perform this disclosure are described (including, for example, references to...). Figures 1 to 6 The described techniques and methods are used in an example processing system 800. In some aspects, the processing system 800 may correspond to an inference system (such as the one mentioned above). Figure 2 and / or Figure 4 (Discussion of the inference system). For example, processing system 800 may correspond to a system that uses a quantized machine learning model to generate inference during runtime. Although depicted as a single system for clarity of concept, in some respects, as discussed above, the operations described below with respect to processing system 800 can be distributed across any number of devices or systems.

[0140] The processing system 800 includes a central processing unit (CPU) 802, which in some examples may be a multi-core CPU. Instructions executed at the CPU 802 may be loaded, for example, from program memory associated with the CPU 802 or from a memory partition (e.g., a partition of memory 824).

[0141] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, a multimedia component 810 (e.g., a multimedia processing unit), and a wireless connectivity component 812.

[0142] NPUs (such as the NPU 808) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0143] NPUs such as the NPU 808 are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0144] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0145] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0146] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0147] In some implementations, the NPU 808 is part of one or more of the CPU 802, GPU 804, and / or DSP 806.

[0148] In some examples, the wireless connectivity component 812 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and / or other wireless data transmission standards. The wireless connectivity component 812 is further coupled to one or more antennas 814.

[0149] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0150] The processing system 800 may also include one or more input and / or output devices 822, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0151] In some examples, one or more processors in the processing system 800 may be based on the ARM or RISC-V instruction set.

[0152] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0153] Specifically, in this example, memory 824 includes a decomposition component 824A, a multiplication component 824B, a dequantization component 824C, and an aggregation component 824D. Memory 824 also includes components for one or more models (e.g., Figure 1 Quantitative machine learning model 145, which includes Figure 1 The model parameters 824E are the quantized parameter subtensor 135. Although not included in the illustrated example, in some respects, memory 824 may also include other data, such as input data (e.g., to be used as input to the model). Although for the sake of conceptual clarity, Figure 8 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) may be implemented together or separately.

[0154] The processing system 800 also includes a decomposition circuit 826, a multiplication circuit 827, a dequantization circuit 828, and a aggregation circuit 829. The depicted circuits and other circuits not depicted can be configured to perform various aspects of the techniques described herein.

[0155] For example, decomposition component 824A and / or decomposition circuit 826 (which may correspond to Figure 2 The decomposition component 210 can be used to decompose the input tensor using outlier indices, as discussed above. For example, the decomposition component 824A and / or the decomposition circuit 826 can generate input subtensors based on outlier indices (e.g., having a first subtensor that includes input elements corresponding to the outlier indices and a second subtensor that includes input elements corresponding to the remaining indices or values).

[0156] Multiplication component 824B and / or multiplication circuit 827 (which may correspond to) Figure 2 The multiplication component 220 can be used to multiply an input subtensor with a (quantized) parameter subtensor, as discussed above. For example, multiplication component 824B and / or multiplication circuit 827 can perform matrix multiplication between each input subtensor and the corresponding (quantized) parameter subtensor to generate a corresponding output subtensor.

[0157] Dequantization component 824C and / or dequantization circuit 828 (which may correspond to Figure 2 The dequantization component 230 can be used to dequantize the output subtensor, as discussed above. For example, the dequantization component 824C and / or the dequantization circuit 828 can use the quantization parameters used to quantize each given quantization parameter subtensor to dequantize the corresponding output subtensor.

[0158] Polymerization component 824D and / or polymerization circuit 829 (which may correspond to) Figure 2The aggregation component 240 can be used to aggregate dequantized output subtensors, as discussed above. For example, the aggregation component 824D and / or the aggregation circuit 829 can aggregate output subtensors by performing element-wise summation to generate a total output tensor for operation.

[0159] Although for clarity Figure 8 While depicted as separate components and circuits, the decomposition circuit 826, multiplication circuit 827, dequantization circuit 828, and aggregation circuit 829 may be implemented jointly or separately in other processing devices of the processing system 800, such as in the CPU 802, GPU 804, DSP 806, NPU 808, etc.

[0160] Generally speaking, the processing system 800 and / or its components can be configured to perform the methods described herein.

[0161] It is worth noting that, in other aspects, such as when the processing system 800 is a server computer, components of the processing system 800 may be omitted. For example, in other aspects, the multimedia component 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation processor 820 may be omitted. Furthermore, various aspects of the processing system 800 may be distributed among multiple devices.

[0162] Example Terms

[0163] Specific implementation examples are described in the following numbered clauses: Clause 1: A method comprising: accessing a parameter tensor of a machine learning model; identifying a set of rows in the parameter tensor, each including one or more outliers; decomposing the parameter tensor into a first parameter sub-tensor corresponding to the set of rows and a second parameter sub-tensor corresponding to at least one remaining row in the parameter tensor; quantizing the first parameter sub-tensor according to a first quantization scheme; quantizing the second parameter sub-tensor according to a second quantization scheme; and generating a quantized version of the machine learning model including the quantized first parameter sub-tensor and the quantized second parameter sub-tensor.

[0164] Clause 2: The method according to Clause 1, wherein: the first quantization scheme includes a first quantization bit width, and the second quantization scheme includes a second quantization bit width smaller than the first quantization bit width.

[0165] Clause 3: The method according to any one of Clauses 1 to 2, the method further comprising identifying the one or more outliers, including: determining one or more outlier criteria; and using the one or more outlier criteria to evaluate each corresponding value in the parameter tensor.

[0166] Clause 4: The method described in Clause 3, wherein the one or more outlier criteria includes a magnitude threshold.

[0167] Clause 5: The method according to any one of Clauses 1 to 4 further includes generating an outlier index set comprising a corresponding index of each corresponding row in the row set, wherein the outlier index set is used to decompose the input tensor during inference.

[0168] Clause 6: The method according to Clause 5, wherein the inference comprises: accessing an input tensor; decomposing the input tensor into a first input sub-tensor having columns corresponding to the outlier index set and a second input sub-tensor corresponding to at least one remaining column of the input tensor; generating a first output sub-tensor based on multiplying the first input sub-tensor by the first parameter sub-tensor; and generating a second output sub-tensor based on multiplying the second input sub-tensor by the second parameter sub-tensor.

[0169] Clause 7: The method according to any one of Clauses 5 to 6, wherein the inference further comprises: dequantizing the first output subtensor and the second output subtensor; and generating an output tensor by summing the dequantized first output subtensor and the dequantized second output subtensor element by element.

[0170] Clause 8: The method according to any one of Clauses 1 to 7, wherein the parameter tensor includes weights for matrix multiplication operations.

[0171] Clause 9: The method according to Clause 8, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

[0172] Item 10: A method comprising: accessing an input tensor of a layer of a machine learning model; decomposing the input tensor into a first input sub-tensor corresponding to a set of outlier indices and a second input sub-tensor corresponding to at least one remaining element in the input tensor; generating a first output sub-tensor based on multiplying the first input sub-tensor by a first parameter sub-tensor; generating a second output sub-tensor based on multiplying the second input sub-tensor by a second parameter sub-tensor; and generating an output tensor of the layer of the machine learning model based on the first output sub-tensor and the second output sub-tensor.

[0173] Clause 11: The method according to Clause 10, wherein generating the output tensor comprises: dequantizing the first output sub-tensor and the second output sub-tensor; and summing the dequantized first output sub-tensor and the dequantized second output sub-tensor element by element.

[0174] Clause 12: The method according to any one of Clauses 10 to 11, wherein: the first parameter subtensor is quantized according to a first quantization scheme, and the second parameter subtensor is quantized according to a second quantization scheme.

[0175] Clause 13: The method according to Clause 12, wherein: the first quantization scheme includes a first quantization bit width, and the second quantization scheme includes a second quantization bit width greater than the first quantization bit width.

[0176] Clause 14: The method according to any one of Clauses 10 to 13, wherein the outlier index set indicates a set of rows in the non-quantized parameter tensors of the layers of the machine learning model that each includes one or more outliers.

[0177] Clause 15: The method according to any one of Clauses 10 to 14, wherein the first parameter subtensor and the second parameter subtensor include weights for matrix multiplication operations.

[0178] Clause 16: The method according to Clause 15, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

[0179] Clause 17: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 16.

[0180] Clause 18: A processing system comprising components for performing the method according to any one of Clauses 1 to 16.

[0181] Clause 19: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 16.

[0182] Clause 20: A non-transitory computer-readable medium encoding logic, which, when executed by a processing system, causes the processing system to perform the method according to any one of Clauses 1 to 16.

[0183] Clause 21: An apparatus comprising logic circuitry configured to perform the method according to any one of Clauses 1 to 16.

[0184] Clause 22: A computer program product embodied on a computer-readable storage medium, said computer-readable storage medium including code for performing a method according to any one of Clauses 1 to 16.

[0185] Additional Notes

[0186] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of the disclosure herein may be embodied by one or more elements of these claims.

[0187] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0188] As used in this article, the phrase “at least one of” in a list of items refers to any combination of those items (including a single member). As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0189] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include parsing, selecting, picking, building, etc.

[0190] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0191] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly referred to by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known later, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: One or more memories, the one or more memories including processor-executable instructions; and One or more processors, the one or more processors being configured to execute processor-executable instructions and enable the processing system: Access the parameter tensors of a machine learning model; Each of the rows in the parameter tensor contains one or more outlier values; The parameter tensor is decomposed into a first parameter sub-tensor corresponding to the row set and a second parameter sub-tensor corresponding to at least one remaining row in the parameter tensor. The first parameter sub-tensor is quantized according to the first quantization scheme; The second parameter subtensor is quantized according to the second quantization scheme; and Generate a quantized version of the machine learning model that includes the quantized first parametric subtensor and the quantized second parametric subtensor.

2. The processing system according to claim 1, wherein: The first quantization scheme includes a first quantization bit width, and The second quantization scheme includes a second quantization bit width that is smaller than the first quantization bit width.

3. The processing system of claim 1, wherein the one or more processors are configured to further execute processor-executable instructions to cause the processing system to identify the one or more outliers, wherein, In order to identify the one or more outliers, the one or more processors are configured to execute processor-executable instructions to enable the processing system to: Identify one or more outlier criteria; as well as Use one or more outlier criteria to evaluate each corresponding value in the parameter tensor.

4. The processing system according to claim 3, wherein the one or more outlier criteria include a magnitude threshold.

5. The processing system according to claim 1, wherein, The one or more processors are configured to further execute processor-executable instructions to cause the processing system to generate an outlier index set including a corresponding index of each corresponding row in the row set, wherein the outlier index set is used to decompose the input tensor during inference.

6. The processing system of claim 5, wherein, for inference purposes, one or more second processors of the inference system are configured to execute second processor-executable instructions to cause the inference system to: Access the input tensor; The input tensor is decomposed into a first input sub-tensor having columns corresponding to the outlier index set and a second input sub-tensor corresponding to at least one of the remaining columns in the input tensor; The first output subtensor is generated by multiplying the first input subtensor with the first parameter subtensor; and The second output subtensor is generated by multiplying the second input subtensor with the second parameter subtensor.

7. The processing system of claim 6, wherein, for inference purposes, the one or more second processors of the inference system are configured to execute second processor-executable instructions to cause the inference system to: Dequantize the first output subtensor and the second output subtensor; and The output tensor is generated by summing the first output sub-tensor and the second output sub-tensor of the dequantization element by element.

8. The processing system of claim 1, wherein the parameter tensor includes weights for matrix multiplication operations.

9. The processing system of claim 8, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

10. A processing system, the processing system comprising: One or more memories, the one or more memories including processor-executable instructions; and One or more processors, the one or more processors being configured to execute processor-executable instructions and enable the processing system: Access the input tensors of the layers in a machine learning model; The input tensor is decomposed into a first input sub-tensor corresponding to the set of outlier indexes and a second input sub-tensor corresponding to at least one remaining element in the input tensor. The first output subtensor is generated by multiplying the first input subtensor with the first parameter subtensor. The second output subtensor is generated by multiplying the second input subtensor with the second parameter subtensor. as well as The output tensor of the layer of the machine learning model is generated based on the first output sub-tensor and the second output sub-tensor.

11. The processing system according to claim 10, wherein, Generating the output tensor includes the one or more processors being configured to further execute processor-executable instructions to enable the processing system to: Dequantize the first output subtensor and the second output subtensor; and The first output subtensor and the second output subtensor after dequantization are summed element by element.

12. The processing system according to claim 10, wherein: The first parameter sub-tensor is quantized according to the first quantization scheme, and The second parameter subtensor is quantized according to the second quantization scheme.

13. The processing system according to claim 12, wherein: The first quantization scheme includes a first quantization bit width, and The second quantization scheme includes a second quantization bit width that is greater than the first quantization bit width.

14. The processing system of claim 10, wherein the outlier index set indicates a set of rows in the non-quantized parameter tensors of the layers of the machine learning model that each includes one or more outliers.

15. The processing system of claim 10, wherein the first parameter subtensor and the second parameter subtensor include weights for matrix multiplication operations.

16. The processing system of claim 15, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

17. A processor-implemented method, the processor-implemented method comprising: Access the parameter tensors of a machine learning model; Each of the rows in the parameter tensor contains one or more outlier values; The parameter tensor is decomposed into a first parameter sub-tensor corresponding to the row set and a second parameter sub-tensor corresponding to at least one remaining row in the parameter tensor. The first parameter sub-tensor is quantized according to the first quantization scheme; The second parameter subtensor is quantized according to the second quantization scheme; and Generate a quantized version of the machine learning model that includes the quantized first parametric subtensor and the quantized second parametric subtensor.

18. The processor-implemented method according to claim 17, wherein: The first quantization scheme includes a first quantization bit width, and The second quantization scheme includes a second quantization bit width that is smaller than the first quantization bit width.

19. The processor-implemented method of claim 17, further comprising identifying the one or more outliers, including: Identify one or more outlier criteria; as well as Use one or more outlier criteria to evaluate each corresponding value in the parameter tensor.

20. The processor-implemented method of claim 19, wherein the one or more outlier criteria include a magnitude threshold.

21. The processor-implemented method of claim 17, further comprising generating an outlier index set including a corresponding index of each corresponding row in the row set, wherein the outlier index set is used to decompose the input tensor during inference.

22. The processor-implemented method of claim 21, wherein the inference includes: Access the input tensor; The input tensor is decomposed into a first input sub-tensor having columns corresponding to the outlier index set and a second input sub-tensor corresponding to at least one of the remaining columns in the input tensor; The first output subtensor is generated by multiplying the first input subtensor with the first parameter subtensor. as well as The second output subtensor is generated by multiplying the second input subtensor with the second parameter subtensor.

23. The processor-implemented method of claim 22, wherein the inference further comprises: Dequantize the first output sub-tensor and the second output sub-tensor; as well as The output tensor is generated by summing the first output sub-tensor and the second output sub-tensor of the dequantization element by element.

24. The processor-implemented method of claim 17, wherein the parameter tensor includes weights for matrix multiplication operations.

25. The processor-implemented method of claim 24, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

26. A processor-implemented method, the processor-implemented method comprising: Access the input tensors of the layers in a machine learning model; The input tensor is decomposed into a first input sub-tensor corresponding to the set of outlier indexes and a second input sub-tensor corresponding to at least one remaining element in the input tensor. The first output subtensor is generated by multiplying the first input subtensor with the first parameter subtensor. The second output subtensor is generated by multiplying the second input subtensor with the second parameter subtensor. as well as The output tensor of the layer of the machine learning model is generated based on the first output sub-tensor and the second output sub-tensor.

27. The processor-implemented method of claim 26, wherein generating the output tensor comprises: Dequantize the first output sub-tensor and the second output sub-tensor; as well as The first output subtensor and the second output subtensor after dequantization are summed element by element.

28. The processor-implemented method according to claim 26, wherein: The first parameter sub-tensor is quantized according to the first quantization scheme, and The second parameter subtensor is quantized according to the second quantization scheme.

29. The processor-implemented method according to claim 28, wherein: The first quantization scheme includes a first quantization bit width, and The second quantization scheme includes a second quantization bit width that is greater than the first quantization bit width.

30. The processor-implemented method of claim 26, wherein the outlier index set indicates a set of rows in the non-quantized parameter tensors of the layers of the machine learning model that each includes one or more outliers.

31. The processor-implemented method of claim 26, wherein the first parameter subtensor and the second parameter subtensor include weights for matrix multiplication operations.

32. The processor-implemented method of claim 31, wherein the matrix multiplication operation is performed as part of at least one of: a feedforward operation of the machine learning model, or an attention operation of the machine learning model.

Images