Mixed-precision quantization of machine learning model parameters

Abstract

Techniques and apparatus for improving the quantization of machine learning models are disclosed. The parameter tensor of a machine learning model is accessed, and a set of rows is identified in the parameter tensor, each containing one or more outliers. The parameter tensor is decomposed into a first parameter subtensor corresponding to the set of rows and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor. The first parameter subtensor is quantized according to a first quantization scheme, and the second parameter subtensor is quantized according to a second quantization scheme. A quantized version of the machine learning model is generated, containing the quantized first and second parameter subtensors.

Description

Technical Field

[0001]

[0001] Aspects of the present disclosure relate to machine learning.

[0002]

[0002] A wide variety of machine learning model architectures have become widespread and are used to provide solutions for numerous prediction problems. Although particular architectures may vary, machine learning models generally rely on a set of model parameters having values that are learned or trained based on training data, which may include labeled data and / or unlabeled data. Many architectures (e.g., deep learning models) use a large number of such parameters (in some cases, up to billions) to provide better utility. Furthermore, in many cases larger models (e.g., models with more parameters) tend to perform better (e.g., with higher prediction accuracy) and / or tend to be better suited for more complex prediction tasks. However, even relatively small models generally have a relatively large number of parameters and a considerable memory footprint.

[0003]

[0003] Such a large number of parameters inherently results in a considerable memory and / or storage footprint, as well as similarly substantial usage of other computing resources. Model size has become particularly problematic in resource-constrained scenarios, where it is desired to deploy a trained model on a device with relatively limited resources (e.g., mobile devices, embedded devices, smart vehicles, etc.). Some conventional approaches to address such concerns involve parameter quantization. However, parameter quantization is an approximation-based process that inherently introduces errors into the model.

Summary of Invention

[0004]

[0004] A particular embodiment provides a method that includes accessing a parameter tensor of a machine learning model; identifying a set of rows in the parameter tensor, each containing one or more outliers; decomposing the parameter tensor into a first parameter subtensor corresponding to a set of rows and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor; quantizing the first parameter subtensor according to a first quantization scheme; quantizing the second parameter subtensor according to a second quantization scheme; and generating a quantized version of the machine learning model containing the quantized first and second parameter subtensors.

[0005]

[0005] A particular embodiment provides a method that includes: accessing an input tensor for a layer of a machine learning model; decomposing the input tensor into a first input subtensor corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor; generating a first output subtensor based on multiplying the first input subtensor by a first parameter subtensor; generating a second output subtensor based on multiplying the second input subtensor by a second parameter subtensor; and generating an output tensor for a layer of a machine learning model based on the first and second output subtensors.

[0006]

[0006] Another embodiment provides a processing system configured to carry out the methods described above and the methods described herein; a non-temporary computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to carry out the methods described above and the methods described herein; a computer program product embodied on a computer-readable storage medium containing code for carrying out the methods described above and the methods described herein; and a processing system comprising means for carrying out the methods described above and the methods described herein.

[0007]

[0007] The following description and related drawings describe in detail certain exemplary features of one or more embodiments. [Brief explanation of the drawing]

[0008]

[0008] The attached figures illustrate certain features of one or more aspects of the present disclosure and should therefore not be considered to limit the scope of the present disclosure. [Figure 1]

[0009] This is an exemplary workflow for machine learning model parameter quantization according to various aspects of the present disclosure. [Figure 2]

[0010] This is an exemplary workflow for generating inference using quantized machine learning model parameters, according to various aspects of the present disclosure. [Figure 3]

[0011] This flowchart illustrates exemplary methods for generating quantized machine learning models according to various aspects of the present disclosure. [Figure 4]

[0012] This flowchart illustrates exemplary methods for generating output tensors based on quantized machine learning model parameters, according to various aspects of the present disclosure. [Figure 5]

[0013] This flowchart illustrates exemplary methods for parameter quantization according to various aspects of the present disclosure. [Figure 6]

[0014] This flowchart illustrates exemplary methods for generating an output tensor according to various aspects of the present disclosure. [Figure 7]

[0015] This figure shows an exemplary processing system configured to implement various aspects of this disclosure. [Figure 8]

[0016] This figure shows an exemplary processing system configured to implement various aspects of this disclosure.

[0009]

[0017] For ease of understanding, the same reference numeral is used to designate identical elements common to the drawings, where possible. It has been conceived that elements and features of one embodiment can be usefully incorporated into other embodiments without further detail. [Modes for carrying out the invention]

[0010]

[0018] Aspects of this disclosure provide apparatus, methods, processing systems, and computer-readable media for machine learning model parameter quantization.

[0011]

[0019] Quantizing machine learning model parameters is generally a lossy compression technique that involves mapping high-precision parameters (e.g., weights encoded in floating-point representation) to a relatively small set of values ​​that can be represented using fewer bits. For example, 16-bit weights (e.g., weights encoded or stored using 16 bits per weight) can be quantized to 4-bit or 8-bit representations (e.g., quantized weights encoded or stored using 4 bits or 8 bits per weight, respectively), substantially reducing the model's memory footprint. One approach to quantization involves using a scale factor and a zero point (also called a bias and / or offset in some embodiments) to map the original values ​​to values ​​that can be stored precisely with a desired bit width (smaller than the original bit width).

[0012]

[0020] In some embodiments, the bit width is selected as a hyperparameter of the quantization process. In some embodiments, the bit width is selected or determined based on the value of the parameter being quantized (e.g., values ​​with a larger standard deviation are assigned a higher quantized bit width). In some embodiments, several quantization parameters (e.g., zero point and scale factor) may be similarly determined or learned based on the original values ​​of the parameters. For example, based on the mean and standard deviation of the original parameters, various quantization parameters may be selected to map the original values ​​to a set of quantized values ​​with mean and standard deviations that fit within a desired quantized bit width. However, since quantization is inherently an approximation-based approach, errors are naturally introduced by these approaches.

[0013]

[0021] Furthermore, if parameters contain extreme values ​​(e.g., outliers), model accuracy can be substantially reduced by quantization. For example, some conventional approaches involve pruning or removing such outlier parameters before quantization. This may allow for smaller bit-width quantization, but introduces additional errors through the loss of these parameter values. Some conventional approaches involve retaining outliers when quantizing. However, this can introduce considerable quantization loss, as the quantization parameters are chosen to encompass these outliers. To mitigate such unacceptable quantization loss, some conventional approaches instead rely on higher bit-width quantization, which increases the model's memory footprint.

[0014]

[0022] Aspects of this disclosure provide mixed-precision quantization that enables efficient parameter quantization using multiple quantization schemes to minimize (or at least reduce) memory usage while also accommodating outlier parameter values. In some aspects, outliers may be quantized using a first quantization scheme selected to precisely quantize such outliers, while non-outliers may be quantized using a second quantization scheme selected to reduce the quantized parameter size to a minimum (or at least reduced) amount. For example, in some aspects, parameters corresponding to outliers may be quantized to a first quantized bit width, while the remaining parameters may be quantized to a second quantized bit width smaller than the first quantized bit width. While some embodiments described herein illustrate the use of two bit widths (a first bit width for outlier parameters and a second bit width for non-outliers), in some embodiments, three or more bit widths may be used (e.g., a first bit width for non-outlier parameters, a second bit width for extreme outlier parameters, and a third bit width (between the first and second bit widths) for moderate outliers that are not as extreme as the extreme outlier parameters).

[0015]

[0023] Several aspects of this disclosure are used to quantize parameters used in matrix multiplication operations. Matrix multiplication, in which an input tensor (e.g., a matrix of activation data from previous layers in a model) is multiplied by a parameter tensor (e.g., a matrix of weights), is a common operation in various machine learning model architectures. For example, feedforward layers or feedforward operations are often implemented using matrix multiplication. Similarly, transformers (with self-attention) generally use multiple matrix multiplication operations to produce an attention output. In many conventional architectures, the computational cost (e.g., memory and / or processing resources) used to perform inference is primarily caused by matrix multiplication operations. Therefore, in some aspects, the parameter tensor (e.g., a weight matrix) used in matrix multiplication operations may be decomposed based on identified outlier parameter values, and each subtensor may be quantized using appropriate quantization parameters to balance reduction of parameter size with preservation of model accuracy.

[0016]

[0024] In some embodiments, during inference, the input tensor to an operation (e.g., input to matrix multiplication) may be similarly decomposed based on outliers identified in the parameter tensor, and each input subtensor may be multiplied with the corresponding quantized subtensor from the parameter tensor. The resulting intermediate outputs may then be aggregated or combined (e.g., using element-wise sums) to generate an output tensor for the matrix multiplication operation.

[0017] Exemplary Workflow for Quantizing Machine Learning Model Parameters

[0025] FIG. 1 shows an example workflow 100 for machine learning model parameter quantization. In some aspects, workflow 100 is performed by a quantization system (e.g., a computing system that performs model quantization on a trained machine learning model). For example, the quantization system may be a component of a training system (e.g., a component of a computing system that trained the machine learning model), a component of an inference system (e.g., a component of a computing system that uses the trained machine learning model for inference), or an independent system separate from the training system and the inference system.

[0018]

[0026] The illustrated workflow 100 is implemented by an outlier component 110, a decomposition component 120, a quantization component 130, and a compilation component 140. Although shown as separate components for conceptual clarity, in some aspects, the operations of the illustrated components (and other components not illustrated) may be combined or distributed across any number and type of components and systems. Furthermore, the illustrated components may generally be implemented using hardware, software, or a combination of hardware and software.

[0019]

[0027] In the illustrated example, parameter tensor 105 is accessed by outlier component 110, which generates outlier index 115. Parameter tensor 105 generally corresponds to a subset of parameters of a machine learning model. That is, parameter tensor 105 may include one or more parameters having values learned during a training process of the model (e.g., based on supervised training and / or unsupervised training). For example, in some aspects, parameter tensor 105 includes a weight matrix for a matrix multiplication operation. In some aspects, parameter tensor 105 generally corresponds to an original unquantized set or an unquantized set of parameters from an unquantized machine learning model.

[0020]

[0028] In the illustrated embodiment, outlier index 115 indicates an index (in parameter tensor 105) corresponding to a parameter having an outlier. As used herein, an "outlier" parameter refers to a parameter having a value that satisfies one or more outlier criteria. In general, the specific definition of outlier criteria may vary depending on the particular implementation. For example, in some aspects, a parameter is an outlier if its value exceeds a defined threshold magnitude (e.g., a defined threshold above or below zero). In some aspects, a parameter is an outlier if the parameter is outside a defined range from the average value of values in parameter tensor 105. For example, a given parameter may be classified as an outlier if its value is more than two standard deviations away from the average value of values in parameter tensor 105. In general, any suitable criteria may be used to define outliers.

[0021]

[0029] In some aspects, outlier index 115 explicitly indicates the index (e.g., row and column of the outlier) of each outlier. In some aspects, the outlier index is a row index indicating the row or rows having one or more outliers and / or a column index indicating the column or columns having one or more outliers. For example, if the second row of parameter tensor 105 includes one or more outliers, outlier index 115 may indicate a row index of "2" as corresponding to the one or more outliers. In particular, one or more other values in the same row (e.g., in the second row) may be non-outliers. For example, the second row may include fifteen non-outlier values and one outlier value. In some aspects, outlier index 115 may indicate that the second row includes at least one outlier regardless of other (non-outlier) values in the row.

[0022]

[0030] In some embodiments, the outlier component 110 may similarly generate a set of non-outlier indices from the parameter tensor 105 that indicate rows(s) that do not contain outliers. In other embodiments, the non-outlier indices can be implicitly inferred, since non-outlier rows essentially correspond to all other parameters or rows not indicated in the outlier index 115. For example, suppose the parameter tensor 105 contains parameters in eight rows, and the first, third, fourth, and eighth rows contain one or more outliers. In some embodiments, the outlier index 115 may indicate the first, third, fourth, and eighth rows. The system can then infer that the remaining rows (the second, fifth, sixth, and seventh rows) do not contain any outliers.

[0023]

[0031] As illustrated, the outlier index 115 is then provided to the compilation component 140, which is described in more detail below. The outlier index 115 is further accessed by the decomposition component 120. In the illustrated embodiment, the decomposition component 120 further accesses the parameter tensor 105. Based on the outlier index 115, the decomposition component 120 decomposes the parameter tensor 105 into a set of parameter subtensors 125A and 125B (collectively, the parameter subtensor 125). As used herein, “decomposing” a tensor (also called partitioning, slicing, or defining in some embodiments) refers to distributing the elements of a tensor across two or more subtensors. For example, in the illustrated embodiment, the parameter subtensor 125A may contain any outliers found in the parameter tensor 105 (e.g., as indicated by the outlier index 115), and the parameter subtensor 125B may contain the remaining (non-outlier) elements from the parameter tensor 105.

[0024]

[0032] In some embodiments, as described above, the outlier index 115 indicates which rows (one or more) in the parameter tensor 105 contain at least one outlier. In some such embodiments, the decomposition component 120 slices the parameter tensor 105 to separate the rows (one or more) indicated by the outlier index 115 (each containing at least one outlier) in order to form the parameter subtensor 125A. The decomposition component 120 uses the remaining rows (one or more) (not indicated by the outlier index 115 and not containing outliers) to form the parameter subtensor 125B. That is, in some embodiments, the parameter subtensor 125B is a matrix of parameters (e.g., weights) that do not contain outliers, and the parameter subtensor 125A is a matrix of parameters (e.g., weights) that contain any outliers found in the parameter tensor 105. As described above, the parameter subtensor 125A may further contain one or more non-outliers (for example, if the non-outlier elements are in the same row as the outlier elements in the parameter tensor 105).

[0025]

[0033] The illustrated embodiment shows that the decomposition component 120 decomposes the parameter tensor 105 into two parameter subtensors 125, but in some embodiments, any number of subtensors may be used. For example, in some embodiments, the outlier component 110 may generate multiple sets of indices, such as one set of outlier indices used to indicate values ​​that exceed a first threshold (e.g., weights with values ​​farther than two standard deviations from the mean), and a second set of outlier indices used to indicate values ​​that exceed a second (lower) threshold but not exceed a first (higher) threshold (e.g., weights with values ​​between one and two standard deviations from the mean). In some such embodiments, the decomposition component 120 may generate three parameter subtensors 125, namely, a first parameter subtensor for extreme outliers (e.g., values ​​greater than two standard deviations from the mean), a second parameter subtensor for moderate outliers (e.g., values ​​between one and two standard deviations from the mean), and a third parameter subtensor for the remaining (non-outlier) values.

[0026]

[0034] In the illustrated workflow 100, the parameter subtensor 125 is accessed by each quantization component 130, which generates the corresponding quantized parameter subtensors 135A and 135B (collectively, the quantized parameter subtensor 135). In general, the quantization component 130 may use any number and type of quantization scheme to quantize the parameter subtensor 125. In some embodiments, the quantization component 130 uses a different quantization scheme for each parameter subtensor 125. For example, the quantization component 130 may determine a separate set of quantization parameters for each individual parameter subtensor. As an example, the quantization component 130 may evaluate the values ​​(one or more) contained in a given parameter subtensor 125 to determine a suitable quantization scale factor and / or zero point for the given parameter subtensor. In this way, the quantized parameter subtensor 135A (which may correspond to parameter subtensor 125A) may be quantized using a first scale and zero point particularly selected for the values ​​reflected in parameter subtensor 125A, and the quantized parameter subtensor 135B (which may correspond to parameter subtensor 125B) may be similarly quantized using a second scale and zero point particularly selected for the values ​​reflected in parameter subtensor 125B. In other embodiments, the quantization component uses at least two different quantization schemes for three or more parameter subtensors.

[0027]

[0035] In some embodiments, the quantization component 130 may further select or determine a bit width suitable for each individual parameter subtensor 125. For example, based on a predefined configuration and / or a dynamic evaluation of the individual parameter subtensors 125, the quantization component 130 may determine to use a relatively smaller bit width to quantize parameter subtensor 125B (which does not contain outliers) compared to the bit width used to quantize parameter subtensor 125A (which contains one or more outliers). As described above, using a relatively larger bit width to quantize these outlier subtensors may allow for improved accuracy, while using a relatively smaller bit width for non-outliers (which do not benefit from the improved accuracy of larger bit widths) may minimize (or at least reduce) the total size (e.g., number of bits) of the collectively quantized parameters.

[0028]

[0036] For example, in some embodiments, the quantization hyperparameter may indicate that the parameter subtensor 125A (including outliers) should be quantized to one bit width (e.g., 16 bits), while the parameter subtensor 125B (excluding outliers) should be quantized to a second bit width (e.g., 8 bits). In some embodiments, in addition to or instead of using such pre-configured bit widths, the quantization component 130 may dynamically determine a suitable bit width. For example, based on the range of values ​​and / or standard deviation in a given parameter subtensor 125, the quantization component 130 may select a bit width to balance the precision and size of the resulting quantized parameter subtensor 135.

[0029]

[0037] As described above, two quantized parameter subtensors 135 (generated using two separate quantization schemes) are shown for the sake of clarity of the concept, but in some aspects, the quantization component 130 can generate any number of quantized parameter subtensors 135 using any number and type of quantization schemes. For example, if three parameter subtensors 125 are generated as described above, the quantization component 130 can generate three corresponding quantized parameter subtensors 135 (using, for example, the maximum bit width of the parameter subtensor 125 containing the most extreme outliers, the intermediate bit width of the parameter subtensor 125 containing moderate outliers, and the minimum bit width of the parameter subtensor 125 containing no outliers).

[0030]

[0038] In the illustrated embodiment, the quantized parameter subtensor 135 is then accessed by the compile component 140. The compile component 140 further accesses the outlier index 115 (generated by the outlier component 110) to generate the quantized machine learning model 145. For example, the compile component 140 may aggregate, combine, or compile the quantized parameter subtensor 135 and the outlier index 115 as a machine learning model so that other systems can use the model for inference. In some embodiments, as will be described in more detail below, the outlier index 115 may be included in the quantized machine learning model 145 when the outlier index 115 is used to decompose the input tensor during inference, or otherwise provided to the inference system. For example, the outlier index 115 may be included as a hyperparameter for a portion (one or more) of the model (e.g., a layer or other operation) corresponding to the original parameter tensor 105. In some embodiments, in addition to outlier indices, the compiled component 140 may also include the quantization parameters of each quantized parameter subtensor 135 in the compiled quantized machine learning model 145.

[0031]

[0039] Although not included in the illustrated embodiments, in some embodiments, the compilation component 140 may similarly compile the remaining parameters and / or hyperparameters of the machine learning model to generate the quantized machine learning model 145. For example, workflow 100 may be performed for any other matrix multiplication operations, and for each such matrix multiplication operation, the resulting quantized parameter subtensor 135 and outlier index 115 may be compiled to form a single quantized machine learning model 145. In some embodiments, other parameters and / or hyperparameters from the machine learning model (e.g., parameters not corresponding to matrix multiplication operations) may similarly be compiled to form the quantized machine learning model 145. In some embodiments, such other parameters may preferably similarly be quantized (e.g., using workflow 100 or any other suitable technique).

[0032]

[0040] In this way, the quantized machine learning model 145 can represent or correspond to the original machine learning model (from which the parameter tensor 105 is derived), but its size is substantially reduced due to the reduction in the bit width(s) used to encode the parameters. This reduces the model's memory footprint and the bandwidth consumed by transmitting all or part of the model over communication links (for example, over a network such as the Internet to deploy the model to an inference system, and over links used to retrieve parameters from memory during inference and send the retrieved parameters to a processing unit). This significantly improves the performance of the computing devices involved.

[0033] Exemplary workflow for generating inference using quantized machine learning model parameters

[0041] Figure 2 shows an exemplary workflow 200 for generating inference using quantized machine learning model parameters. In some embodiments, the workflow 200 is performed by an inference system (e.g., a computing system that uses a machine learning model trained for inference). For example, the inference system may be a separate and independent system from the training system (e.g., a system that trains the machine learning model) and / or the quantization system (e.g., a system that performs model quantization on the trained machine learning model). In other embodiments, the inference system may be implemented as a component of another system, such as a component of the training system and / or a component of the quantization system.

[0034]

[0042] The illustrated workflow 200 is implemented by a decomposition component 210, a set of multiplication components 220A and 220B (collectively, a multiplication component 220), a set of inverse quantization components 230A and 230B (collectively, an inverse quantization component 230), and an aggregation component 240. Although shown as separate components for clarity of concept, in some embodiments, the operation of the illustrated components (and other components not shown) may be combined or distributed across any number and types of components and systems. Furthermore, the illustrated components may generally be implemented using hardware, software, or a combination of hardware and software.

[0035]

[0043] In the illustrated embodiment, the input tensor 205 is accessed by a decomposition component 210, which in turn accesses a set of outlier indices (e.g., outlier index 115 as illustrated). The input tensor 205 generally corresponds to a set of values ​​used as input to a layer or operation of a machine learning model. For example, the input tensor 205 may contain activation data output by an activation function of a previous layer, or (e.g., if the operation is in the first layer) the input tensor 205 may contain feature data used as input to the model. In some embodiments, the input tensor 205 is received as input to a matrix multiplication operation (e.g., for feedforward and / or attention operations), as described above.

[0036]

[0044] The outlier index 115 generally indicates the index of any outlier in the parameter tensor (e.g., parameter tensor 105 in Figure 1), as described above. That is, the outlier index 115 may be generated during the quantization of a machine learning model. In some embodiments, the outlier index 115 explicitly indicates the outlier (e.g., the row and column of each outlier in the original parameter tensor). In some embodiments, the outlier index 115 indicates the row(s) in the parameter tensor that has one or more outliers. In some embodiments, as described above, the outlier index 115 is provided as metadata or a hyperparameter associated with a matrix multiplication operation in the model.

[0037]

[0045] In the illustrated embodiment, the decomposition component 210 slices or decomposes the input tensor 205 based on the outlier index 115 to generate a set of input subtensors 215A and 215B (collectively, the input subtensor 215). For example, if the outlier index 115 indicates a column(s) of the original parameter tensor that contains one or more outliers, the decomposition component 210 may slice the input tensor 205 such that the corresponding row(s) of the input tensor 205 are assigned to input subtensor 215A, while the remaining rows (corresponding to the column of parameters not indicated by the outlier index 115 and therefore not having outliers) are assigned to input subtensor 215B.

[0038]

[0046] As another example, if outlier index 115 indicates a row(s) of the original parameter tensor that contains one or more outliers, the decomposition component 210 may slice the input tensor 205 such that the corresponding column(s) of the input tensor 205 are assigned to the input subtensor 215A, while the remaining columns (corresponding to the rows of parameters not indicated by outlier index 115 and therefore not having outliers) are assigned to the input subtensor 215B. For example, if outlier index 115 indicates that the first, third, and fourth rows of the parameter tensor contained outliers, the decomposition component 210 may assign the first, third, and fourth columns of the input tensor 205 to the input subtensor 215A. In this way, by performing column-wise slicing (while the parameter tensor is sliced ​​row by row), the decomposition component 210 ensures that the width of the input subtensor 215 matches the height of the corresponding quantized parameter subtensor, and that each value in the input tensor 205 remains aligned with the corresponding parameter in the parameter tensor, in order to enable matrix multiplication.

[0039]

[0047] As described above, two input subtensors 215 are shown to clarify the concept, but in various embodiments, the decomposition component 210 can slice the input tensor 205 into any number of subtensors (based on outlier index 115). As illustrated, the input subtensors 215 are accessed by their respective multiplication components 220. Specifically, input subtensor 215A is accessed by multiplication component 220A, and input subtensor 215B is accessed by multiplication component 220B. Although two multiplication components 220 (one for each input subtensor 215) are shown to clarify the concept, in some embodiments, the inference system may use a single multiplication component 220 (e.g., processing the input subtensors 215 sequentially). In other embodiments, multiple multiplication components 220 may be used, with at least one of the multiplication components processing multiple input subtensors sequentially.

[0040]

[0048] As illustrated, each multiplication component 220 further accesses the quantized parameter subtensor corresponding to the input subtensor 215 (e.g., the quantized parameter subtensor 135 shown). Specifically, multiplication component 220A accesses the quantized parameter subtensor 135A corresponding to the input subtensor 215A (for example, since both the quantized parameter subtensor 135A and the input subtensor 215A correspond to outlier index 115). In addition, multiplication component 220B accesses the quantized parameter subtensor 135B corresponding to the input subtensor 215B (for example, since both the quantized parameter subtensor 135B and the input subtensor 215B correspond to the remaining values ​​not indicated by outlier index 115).

[0041]

[0049] In the illustrated workflow 200, each of the multiplication components 220 then performs matrix multiplication to multiply each input subtensor 215 by the corresponding quantized parameter subtensor 135. Specifically, multiplication component 220A multiplies the input subtensor 215A by the quantized parameter subtensor 135A to generate the output subtensor 225A. In addition, multiplication component 220B multiplies the input subtensor 215B by the quantized parameter subtensor 135B to generate the output subtensor 225B.

[0042]

[0050] As illustrated, these output subtensors 225A and 225B (collectively, output subtensor 225) are accessed by inverse quantization components 230A and 230B, respectively. Specifically, output subtensor 225A (corresponding to or generated based on outlier parameter values) is accessed by inverse quantization component 230A, and output subtensor 225B (corresponding to non-outlier parameter values) is accessed by inverse quantization component 230B. Two inverse quantization components 230 (one for each output subtensor 225) are shown for clarity, but in some embodiments, the inference system may use a single inverse quantization component 230 (e.g., processing the output subtensors 225 sequentially).

[0043]

[0051] In the illustrated embodiment, the inverse quantization component 230 performs an inverse quantization operation on the output subtensor 225 to generate inverse quantized output subtensors 235A and 235B (collectively, the inverse quantized output subtensor 235). Specifically, the inverse quantization component 230A inverse quantization of the output subtensor 225A based on the quantization parameters used to quantize the quantized parameter subtensor 135A in order to generate the inverse quantized output subtensor 235A, and the inverse quantization component 230B inverse quantization of the output subtensor 225B based on the quantization parameters used to quantize the quantized parameter subtensor 135B in order to generate the inverse quantized output subtensor 235B.

[0044]

[0052] The inversely quantized output subtensor 235 is then accessed by the aggregation component 240, which generates the output tensor 245. In some embodiments, the specific aggregation operations (one or more) used by the aggregation component 240 may vary depending on the specific implementation. For example, the aggregation component 240 may use addition, concatenation, averaging, etc. In the illustrated embodiment, the aggregation component 240 performs an element-wise sum on the inversely quantized output subtensor 235 to generate the output tensor 245.

[0045]

[0053] The output tensor 245 can be used as the output of the matrix multiplication (or other operation) corresponding to the quantized parameter subtensor. For example, the output tensor 245 can be provided as the output of the model and / or as the output to a subsequent layer or operation in a machine learning model (e.g., as part of a feedforward layer or attention component, as described above).

[0046]

[0054] Using workflow 200, the inference system can perform efficient matrix multiplication using mixed-precision quantized parameters. As described above, this mixed-precision representation may represent, or correspond to, the original machine learning model (from which the parameter tensor is derived), and is substantially smaller in size due to the reduced bit width(s) used to encode the parameters. This reduces the model's memory footprint and the bandwidth consumed by transmitting all or part of the model over the communication link (e.g., from memory to the processing unit(s) used to perform the multiplication during inference). This significantly improves the performance of the computing devices involved.

[0047] Exemplary methods for generating quantized machine learning models

[0055] Figure 3 is a flowchart illustrating an exemplary method 300 for generating a quantized machine learning model. In some embodiments, method 300 is carried out by a quantization system such as the quantization system described above with reference to Figure 1.

[0048]

[0056] In block 305, the quantization system accesses a parameter tensor (e.g., parameter tensor 105 in Figure 1). As described above, the parameter sensor generally contains one or more parameters of one or more components or parts of a machine learning model. For example, the parameter tensor may correspond to a weight matrix used to perform matrix multiplication operations within the model (e.g., as part of an attention mechanism or attention operation, or as part of a feedforward layer of a neural network). Generally, in a given model, there can be any number of parameter tensors, each having any number of parameters. In some embodiments, the quantization system may perform method 300 for each such tensor (e.g., sequentially or in parallel) to quantize the model. In some embodiments, as described above, the parameters or elements in the parameter tensor may have values ​​learned or trained during the model training process (e.g., based on labeled and / or unlabeled training data).

[0049]

[0057] In block 310, the quantization system identifies a set of outlier indices in the parameter tensor (e.g., outlier index 115 in Figure 1). In some embodiments, as described above, the outlier index generally indicates the position (one or more) of any parameter (in the tensor) that has an outlier. In some embodiments, outliers are defined with respect to other values ​​in the parameter tensor. For example, a parameter may be designated as an outlier if its value is more than two standard deviations away from the mean of the parameter tensor. In other embodiments, outliers are defined using a fixed threshold (rather than with respect to other values ​​in the tensor). For example, a parameter may be labeled as an outlier if its value exceeds a static threshold (e.g., a value greater than 10). In general, the quantization system may use a variety of criteria to define outliers.

[0050]

[0058] In some embodiments, as described above, the outlier index is a row index, and each indicated row contains at least one outlier parameter value. That is, if the parameter tensor is a weight matrix, the outlier index may indicate one or more rows that have at least one outlier.

[0051]

[0059] In block 315, the quantization system decomposes a parameter tensor based on outlier indices to generate parameter subtensors (e.g., parameter subtensor 125 in Figure 1). For example, in some embodiments, the quantization system creates a first parameter subtensor and a second parameter subtensor. The first parameter subtensor may contain the values ​​indicated in the outlier indices (e.g., by stacking or concatenating the indicated rows from the parameter tensor to generate a new matrix). The second parameter subtensor may contain the remaining values ​​(e.g., by stacking or concatenating the rows not indicated in the outlier indices from the parameter tensor). As mentioned above, a non-outlier subtensor does not contain outliers, but an outlier subtensor may contain one or more non-outliers (in addition to all outliers from the parameter tensor). Although the decomposition of a parameter tensor into two subtensors is described for clarity of concept, as mentioned above, the quantization system may decompose a parameter tensor into any number of subtensors.

[0052]

[0060] In block 320, the quantization system selects one of the parameter subtensors. Generally, the quantization system processes each subtensor during method 300, so the quantization system can use any suitable technique or criterion to select a subtensor. Furthermore, although shown as an iterative process (where the quantization system sequentially selects and processes each subtensor) for clarity, in some embodiments, the quantization system may process some or all of the subtensors in parallel.

[0053]

[0061] In block 325, the quantization system determines or selects a quantization scheme for the selected subtensor. In some embodiments, as described above, the quantization system determines the quantization scheme by identifying a defined scheme (e.g., shown in a previously generated or defined mapping) corresponding to the selected subtensor. For example, the quantization system may use a first defined scheme for any non-outlier subtensor and a second defined scheme for any subtensor containing outliers. In some embodiments, as described above, the quantization system may determine the quantization scheme based on the distribution of values ​​in the selected subtensor. For example, the quantization system may select or define a quantization scheme based on the mean, range of values, standard deviation of values, minimum and / or maximum values, etc. This may allow the quantization system to dynamically quantize each subtensor using a custom-defined quantization for each subtensor.

[0054]

[0062] In some embodiments, as described above, the quantization scheme generally includes a quantization scale (e.g., selected based on the range of values ​​in the subtensor), a quantization zero point or offset (e.g., selected based on the mean of the values ​​in the subtensor), and a quantization bit width (e.g., selected based on the range of values ​​in the subtensor and / or based on a defined mapping based on whether the subtensor contains outliers). For example, as described above, a quantization system may select a quantization scheme with a relatively high bit width (e.g., using more bits to encode each quantization parameter) for subtensors containing outliers, and a quantization scheme with a relatively low bit width (e.g., using fewer bits to encode each quantization parameter) for non-outlier subtensors.

[0055]

[0063] In block 330, the quantization system quantizes the selected parameter subtensor using the determined quantization scheme (for example, to generate the quantized parameter subtensor 135 in Figure 1). As described above, quantizing a subtensor generally involves encoding each parameter with a relatively smaller bit width compared to the unquantized parameter. For example, for each parameter, the quantization system may (i) multiply the parameter by the determined quantization scale and / or (ii) add or subtract a zero point. The resulting value can then be encoded using the determined bit width. In this way, the quantized subtensor can be represented using relatively fewer bits compared to the unquantized subtensor.

[0056]

[0064] In block 335, the quantization system determines whether there is at least one additional parameter subtensor that has not yet been quantized (generated in block 315). If so, method 300 returns to block 320 to select a parameter subtensor to process. Otherwise, method 300 continues to block 340.

[0057]

[0065] In block 340, the quantization system generates a quantized machine learning model based on the quantized subtensors. For example, as described above, the quantization system may compile the quantized subtensors along with the quantized parameters (one or more) used for each subtensor, and relevant metadata such as outlier indices, to represent the original parameter tensor in the quantized model. In some embodiments, generating a quantized machine learning model similarly includes compiling quantized parameters (and other relevant data) for other parameters (one or more) in the original model (e.g., other parameter tensors and / or other layers or parts of the model).

[0058]

[0066] In this way, the quantization system can dynamically generate a quantized version of the (original) machine learning model, which can be represented or stored using fewer bits. This reduces the memory footprint of the model and also reduces the bandwidth consumed by transmitting the quantized model (e.g., over a network to the inference system and / or from memory to the processing components of the inference system). In this way, the quantized model can be used for inference using reduced computational resources, which can enable more efficient execution and / or use of the model on resource-constrained devices.

[0059] Exemplary method for generating output tensors based on quantized machine learning model parameters

[0067] Figure 4 is a flowchart illustrating an exemplary method 400 for generating an output tensor based on quantized machine learning model parameters. In some embodiments, method 400 is carried out by an inference system, such as the inference system described above with reference to Figure 2.

[0060]

[0068] In block 405, the inference system accesses an input tensor (e.g., input tensor 205). As described above, an input tensor is generally used as input to a layer or part of a machine learning model (e.g., as input to a matrix multiplication operation in a feedforward layer and / or attention layer). As described above, an input tensor is generally a tensor containing one or more elements (e.g., vectors, matrices, or scalar values) of data used as input to an operation (e.g., matrix multiplication) of a machine learning model. For example, if the operation is the first layer of the model, the input tensor may be data provided as input to the model (e.g., sensor data). If the operation is part of an internal layer (e.g., an internal attention operation or feedforward layer of the model), the input tensor may contain data output by the previous layer or operation (e.g., activation data from the previous layer).

[0061]

[0069] In block 410, the inference system determines a set of outlier indices corresponding to the current operation or layer of the model. That is, if an input tensor is received or accessed to be processed using a given layer or operation, the inference system may identify the outlier indices generated for that given layer or operation. In some embodiments, as described above, these outlier indices may be determined during the quantization of the model (e.g., by the quantization system). These outlier indices may be provided as part of the compiled quantized model or included therein.

[0062]

[0070] As described above, in some embodiments, the outlier index indicates a row(s) in the original parameter tensor that contains one or more outlier parameter values. This allows the inference system to identify the element(s) in the input tensor corresponding to the row(s) of the outlier.

[0063]

[0071] In block 415, the inference system decomposes the input tensor into a set of input subtensors (e.g., input subtensor 215 in Figure 2) based on outlier indices. For example, if the outlier indices correspond to specific parameters (e.g., indices in specific columns c and rows r for each outlier), the inference system may use these indices to identify the corresponding input elements (e.g., input elements in the same columns c and rows r, or input elements in the inverse index columns r and rows c). In some embodiments, if the outlier indices are row indices (e.g., each index indicates a row from the original parameter tensor that contained at least one outlier), the inference system may identify the corresponding column for each such row.

[0064]

[0072] For example, if the outlier index indicates the row [2,5,...,r], the inference system may generate an input subtensor from the input tensor that contains the column [2,5,...,r]. In some embodiments, in addition to this outlier subtensor (the input subtensor containing the elements corresponding to the parameters in the outlier parameter subtensor), the inference system similarly generates a non-outlier subtensor containing the remaining elements from the input tensor. Although it is explained that the input tensor is decomposed into two subtensors for clarity, as mentioned above, the inference system can decompose the input tensor into any number of subtensors.

[0065]

[0073] In block 420, the inference system selects one of the input subtensors. Generally, the inference system processes each subtensor during method 400, so the inference system can use any preferred technique or criterion to select a subtensor. Furthermore, although shown as an iterative process (where the inference system sequentially selects and processes each subtensor) for clarity, in some embodiments, the inference system may process some or all of the subtensors in parallel.

[0066]

[0074] In block 425, the inference system accesses the parameter subtensor corresponding to the selected subtensor. For example, if the selected subtensor corresponds to an outlier index, the inference system may access the parameter subtensor corresponding to the outlier index. Similarly, if the selected subtensor contains the remaining elements from the input tensor (elements that do not correspond to outlier indices), the inference system may access the parameter subtensor corresponding to the remaining (non-outlier) parameters. In some embodiments, as described above, the parameter subtensor may be a quantized subtensor (quantized by a quantization component based on a set of quantization parameters). That is, in block 425, the inference system may access the corresponding quantized parameter subtensor (for example, one of the quantized parameter subtensors 135 in Figure 1).

[0067]

[0075] In block 430, the inference system generates an output subtensor based on the selected input subtensor and the corresponding parameter subtensor. For example, in the case of matrix multiplication, the inference system may multiply the selected input subtensor by the corresponding parameter subtensor.

[0068]

[0076] In block 435, the inference system dequantizes the generated output subtensor based on the quantization parameters used to quantize the parameter subtensor (accessed in block 425). In some embodiments, as described above, these quantization parameters may be provided as part of or included in a quantized machine learning model, enabling the inference device to appropriately dequantize the parameters.

[0069]

[0077] In block 440, the inference system determines whether there is at least one additional input subtensor remaining to be processed. If so, method 400 returns to block 420. Otherwise, method 400 continues to block 445.

[0070]

[0078] In block 445, the inference system aggregates output subtensors to generate an output tensor (such as output tensor 245 in Figure 2). For example, in some embodiments, the inference system may calculate the element-wise sum of the output subtensors to generate an output tensor. This output tensor corresponds to the output of the current operation or layer in the machine learning model (e.g., the output of a matrix multiplication operation).

[0071]

[0079] In this way, the inference system can generate machine learning model outputs and / or internal values ​​(e.g., the results of matrix multiplication operations) with substantially reduced computational cost compared to at least some conventional machine learning models. Although not included in the illustrated embodiments, in some embodiments, the inference system may similarly perform method 400 for each other relevant operation or layer in the quantized model (e.g., for each other matrix multiplication operation).

[0072] Exemplary methods for parameter quantization

[0080] Figure 5 is a flowchart illustrating an exemplary method 500 for parameter quantization. In some embodiments, method 500 is carried out by a quantization system such as the quantization system described above with reference to Figure 1 and / or Figure 3.

[0073]

[0081] In block 505, we access the parameter tensor of the machine learning model.

[0074]

[0082] In some embodiments, the parameter tensor includes weights for matrix multiplication operations.

[0075]

[0083] In some embodiments, matrix multiplication operations are performed as part of at least one of either a feedforward operation of a machine learning model or an attention operation of a machine learning model.

[0076]

[0084] In block 510, identify sets of rows in the parameter tensor, each containing one or more outliers.

[0077]

[0085] In some embodiments, method 500 further includes identifying one or more outliers, which includes determining one or more outlier criteria and evaluating each individual value in the parameter tensor using one or more outlier criteria.

[0078]

[0086] In some embodiments, one or more outlier criteria include a magnitude threshold.

[0079]

[0087] In some embodiments, method 500 further includes generating a set of outlier indices, each containing a distinct index for each individual row in the set of rows. The set of outlier indices may be used to decompose the input tensor during inference.

[0080]

[0088] In block 515, the parameter tensor is decomposed into a first parameter subtensor corresponding to a set of rows, and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor.

[0081]

[0089] In block 520, the first parameter subtensor is quantized according to the first quantization scheme.

[0082]

[0090] In block 525, the second parameter subtensor is quantized according to the second quantization scheme.

[0083]

[0091] In some embodiments, the first quantization scheme includes a first quantized bit width, and the second quantization scheme includes a second quantized bit width that is smaller than the first quantized bit width.

[0084]

[0092] In block 530, a quantized version of the machine learning model is generated, which includes quantized first and second parameter subtensors.

[0085]

[0093] In some embodiments, inference includes accessing an input tensor, decomposing the input tensor into a first input subtensor having columns corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining column in the input tensor, generating a first output subtensor based on multiplying the first input subtensor by a first parameter subtensor, and generating a second output subtensor based on multiplying the second input subtensor by a second parameter subtensor.

[0086]

[0094] In some embodiments, the inference further includes dequantizing the first and second output subtensors and generating an output tensor by summing the first and second dequantized output subtensors element by element.

[0087] Example generation of output tensors

[0095] Figure 6 is a flowchart illustrating an exemplary method 600 for generating an output tensor. In some embodiments, method 600 is carried out by an inference system, such as the inference system described above with reference to Figures 2 and / or 4.

[0088]

[0096] In block 605, we access the input tensors of the layers in the machine learning model.

[0089]

[0097] In block 610, the input tensor is decomposed into a first input subtensor corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor.

[0090]

[0098] In some embodiments, a set of outlier indices represents a set of rows in the unquantized parameter tensor of a layer of a machine learning model, each containing one or more outliers.

[0091]

[0099] In block 615, a first output subtensor is generated by multiplying a first input subtensor by a first parameter subtensor.

[0092]

[0100] In block 620, a second output subtensor is generated by multiplying a second input subtensor by a second parameter subtensor.

[0093]

[0101] In some embodiments, the first parameter subtensor is quantized according to a first quantization scheme, and the second parameter subtensor is quantized according to a second quantization scheme.

[0094]

[0102] In some embodiments, the first quantization scheme includes a first quantized bit width, and the second quantization scheme includes a second quantized bit width that is greater than the first quantized bit width.

[0095]

[0103] In some embodiments, the first and second parameter subtensors include weights for matrix multiplication operations.

[0096]

[0104] In some embodiments, matrix multiplication operations are performed as part of at least one of either a feedforward operation of a machine learning model or an attention operation of a machine learning model.

[0097]

[0105] In block 625, the output tensors of the layers of the machine learning model are generated based on the first and second output subtensors.

[0098]

[0106] In some embodiments, generating an output tensor includes dequantizing the first and second output subtensors and summing the first and second dequantized output subtensors element by element.

[0099] Exemplary Processing System for Data Quantization

[0107] In some embodiments, the workflows, techniques, and methods described with reference to Figures 1 to 6 may be implemented on one or more devices or systems. Figure 7 shows an exemplary processing system 700 configured to carry out various embodiments of this disclosure, including, for example, the techniques and methods described with reference to Figures 1 to 6. In some embodiments, the processing system 700 may correspond to a quantization system, such as the quantization system described above with reference to Figure 1 and / or Figure 3. For example, the processing system 700 may correspond to a system that quantizes a machine learning model after training. Although shown as a single system for clarity of concept, in some embodiments, as described above, the operations described below with respect to the processing system 700 may be distributed across any number of devices or systems.

[0100]

[0108] The processing system 700 includes a central processing unit (CPU) 702, which in some embodiments may be a multi-core CPU. Instructions executed by the CPU 702 may be loaded, for example, from program memory associated with the CPU 702, or from a memory partition (for example, a partition of memory 724).

[0101]

[0109] The processing system 700 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, a multimedia component 710 (e.g., a multimedia processing unit), and a wireless connectivity component 712.

[0102]

[0110] NPUs, such as the NPU708, are generally dedicated circuits configured to implement control and arithmetic logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs may also be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0103]

[0111] NPUs such as the NPU708 are configured to accelerate the execution of common machine learning tasks, including image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs can be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, the NPU can be part of a dedicated neural network accelerator.

[0104]

[0112] NPUs can be optimized for training or inference, or in some cases, configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0105]

[0113] NPUs, designed to accelerate training, are generally configured to accelerate the optimization of new models, a highly computationally intensive operation that involves inputting existing datasets (often labeled or tagged), iterating through those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0106]

[0114] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs can be configured to take new data as input and process this data rapidly through a model that has already been trained to generate model outputs (e.g., inferences).

[0107]

[0115] In some implementations, the NPU708 is part of one or more of the CPU702, GPU704, and / or DSP706.

[0108]

[0116] In some embodiments, the wireless connectivity component 712 may include subcomponents relating to, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long-Term Evolution (LTE)), fifth-generation connectivity (e.g., 5G or New Radio (NR))), Wi-Fi connectivity, Bluetooth connectivity, and / or other wireless data transmission standards. The wireless connectivity component 712 is further coupled to one or more antennas 714.

[0109]

[0117] The processing system 700 may also include a navigation processor 720, which may include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISPs) 718 associated with any type of image sensor, and / or satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0110]

[0118] The processing system 700 may also include one or more input and / or output devices 722, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0111]

[0119] In some embodiments, one or more of the processors in the processing system 700 may be based on an ARM or RISC-V instruction set.

[0112]

[0120] The processing system 700 also includes memory 724, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 724 includes a computer executable component that can be run by one or more of the aforementioned processors of the processing system 700.

[0113]

[0121] In particular, in this embodiment, memory 724 includes an outlier component 724A, a decomposition component 724B, a quantization component 724C, and a compilation component 724D. Memory 724 further includes model parameters 724E for one or more models (e.g., the parameter tensor 105 in Figure 1). Although not included in the illustrated embodiment, in some embodiments, memory 724 may also include other data, such as training data (e.g., for training and / or fine-tuning a model(s)). Although shown as separate components in Figure 7 for clarity of concept, the illustrated components (and other components not shown) may be implemented collectively or individually in various embodiments.

[0114]

[0122] The processing system 700 further comprises an outlier circuit 726, a decomposition circuit 727, a quantization circuit 728, and a compilation circuit 729. The illustrated circuits, and other circuits not shown, can be configured to carry out various aspects of the techniques described herein.

[0115]

[0123] For example, the outlier component 724A and / or the outlier circuit 726 (which may correspond to the outlier component 110 in Figure 1) may be used, as described above, to identify outlier parameters in the model parameter 724E and / or to generate an outlier index. For example, the outlier component 724A and / or the outlier circuit 726 may use various outlier criteria to identify outliers and generate an outlier index in one or more parameter tensors that indicates the row(s) containing such outliers(s) or outliers(s).

[0116]

[0124] Decomposition component 724B and / or decomposition circuit 727 (which may correspond to decomposition component 120 in Figure 1) can be used to decompose a parameter tensor based on outlier indices, as described above. For example, decomposition component 724B and / or decomposition circuit 727 can decompose a parameter tensor by generating one subtensor containing one or more rows of the parameter tensor that contain one or more outliers, and generating a second subtensor containing the remaining rows of the parameter tensor.

[0117]

[0125] The quantization component 724C and / or the quantization circuit 728 (which may correspond to the quantization component 130 in Figure 1) can be used to quantize parameter subtensors, as described above. For example, the quantization component 724C and / or the quantization circuit 728 can determine or select a quantization scheme for each parameter subtensor and quantize each individual subtensor according to the individual quantization scheme.

[0118]

[0126] The compilation component 724D and / or compilation circuit 729 (which may correspond to the compilation component 140 in Figure 1) can be used to compile quantized parameter subtensors, as well as associated metadata such as outlier indices and quantization parameters, as described above. For example, the compilation component 724D and / or compilation circuit 729 can aggregate a set of quantized parameter subtensors, along with their corresponding outlier indices and quantization parameters, for each operation in a machine learning model, in order to generate a quantized version of the machine learning model.

[0119]

[0127] Although shown as separate components and circuits in Figure 7 for clarity, the outlier circuit 726, the decomposition circuit 727, the quantization circuit 728, and the compilation circuit 729 may be implemented collectively or individually within other processing devices of the processing system 700, such as within the CPU 702, GPU 704, DSP 706, and NPU 708.

[0120]

[0128] In general, the processing system 700 and / or its components may be configured to carry out the methods described herein.

[0121]

[0129] In particular, in other embodiments, elements of the processing system 700 may be omitted, such as when the processing system 700 is a server computer. For example, the multimedia component 710, the wireless connectivity component 712, the sensor processing unit 716, the ISP 718, and / or the navigation processor 720 may be omitted in other embodiments. Furthermore, embodiments of the processing system 700 may be distributed among multiple devices.

[0122] Exemplary processing system for inference

[0130] In some embodiments, the workflows, techniques, and methods described with reference to Figures 1 to 6 may be implemented on one or more devices or systems. Figure 8 shows an exemplary processing system 800 configured to implement various embodiments of this disclosure, including, for example, the techniques and methods described with reference to Figures 1 to 6. In some embodiments, the processing system 800 may correspond to an inference system, such as the inference system described above with reference to Figure 2 and / or Figure 4. For example, the processing system 800 may correspond to a system that uses a quantized machine learning model to generate inferences during runtime. Although shown as a single system for clarity of concept, in some embodiments, as described above, the operations described below with respect to the processing system 800 may be distributed across any number of devices or systems.

[0123]

[0131] The processing system 800 includes a central processing unit (CPU) 802, which in some embodiments may be a multi-core CPU. Instructions executed by the CPU 802 may be loaded, for example, from program memory associated with the CPU 802, or from a memory partition (e.g., a partition of memory 824).

[0124]

[0132] The processing system 800 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, a multimedia component 810 (e.g., a multimedia processing unit), and a wireless connectivity component 812.

[0125]

[0133] NPUs, such as the NPU808, are generally dedicated circuits configured to implement control and arithmetic logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs may also be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0126]

[0134] NPUs such as the NPU808 are configured to accelerate performance on common machine learning tasks, including image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs can be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other embodiments, the NPU can be part of a dedicated neural network accelerator.

[0127]

[0135] NPUs can be optimized for training or inference, or in some cases, configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0128]

[0136] NPUs, designed to accelerate training, are generally configured to accelerate the optimization of new models, a highly computationally intensive operation that involves inputting existing datasets (often labeled or tagged), iterating through those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0129]

[0137] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs can be configured to take new data as input and process this data rapidly through a model that has already been trained to generate model outputs (e.g., inferences).

[0130]

[0138] In some implementations, the NPU808 is part of one or more of the CPU802, GPU804, and / or DSP806.

[0131]

[0139] In some embodiments, the wireless connectivity component 812 may include subcomponents relating to, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long-Term Evolution (LTE)), fifth-generation connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and / or other wireless data transmission standards. The wireless connectivity component 812 is further coupled to one or more antennas 814.

[0132]

[0140] The processing system 800 may also include a navigation processor 820 which may include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0133]

[0141] The processing system 800 may also include one or more input and / or output devices 822, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0134]

[0142] In some embodiments, one or more of the processors in the processing system 800 may be based on an ARM instruction set or a RISC-V instruction set.

[0135]

[0143] The processing system 800 also includes memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 824 includes a computer executable component that can be run by one or more of the aforementioned processors of the processing system 800.

[0136]

[0144] In particular, in this embodiment, memory 824 includes a decomposition component 824A, a multiplication component 824B, an inverse quantization component 824C, and an aggregation component 824D. Memory 824 further includes model parameters 824E of one or more models (e.g., the quantized machine learning model 145 in Figure 1, which includes the quantized parameter subtensor 135 in Figure 1). Although not included in the illustrated embodiment, in some embodiments, memory 824 may also include other data, such as input data (e.g., used as input to a model(s)). Although shown as separate components in Figure 8 for clarity of concept, the illustrated components (and other components not shown) may be implemented collectively or individually in various embodiments.

[0137]

[0145] The processing system 800 further comprises a decomposition circuit 826, a multiplication circuit 827, an inverse quantization circuit 828, and an aggregation circuit 829. The illustrated circuits, and other circuits not shown, can be configured to carry out various embodiments of the techniques described herein.

[0138]

[0146] For example, the decomposition component 824A and / or the decomposition circuit 826 (which may correspond to the decomposition component 210 in Figure 2) can be used to decompose an input tensor using outlier indices, as described above. For example, the decomposition component 824A and / or the decomposition circuit 826 can generate input subtensors based on outlier indices (for example, using a first subtensor containing input elements corresponding to outlier indices, and a second subtensor containing input elements corresponding to the remaining indices or values).

[0139]

[0147] The multiplication component 824B and / or the multiplication circuit 827 (which may correspond to the multiplication component 220 in Figure 2) can be used to multiply an input subtensor with a (quantized) parameter subtensor, as described above. For example, the multiplication component 824B and / or the multiplication circuit 827 may perform matrix multiplication between each input subtensor and the corresponding (quantized) parameter subtensor to generate the corresponding output subtensor.

[0140]

[0148] The inverse quantization component 824C and / or the inverse quantization circuit 828 (which may correspond to the inverse quantization component 230 in Figure 2) can be used to inverse quantize the output subtensor, as described above. For example, the inverse quantization component 824C and / or the inverse quantization circuit 828 may use the quantization parameters used to quantize each given quantized parameter subtensor to inverse quantize the corresponding output subtensor.

[0141]

[0149] The aggregation component 824D and / or aggregation circuit 829 (which may correspond to the aggregation component 240 in Figure 2) can be used to aggregate the inversely quantized output subtensors, as described above. For example, the aggregation component 824D and / or aggregation circuit 829 can aggregate the output subtensors by performing element-wise summations to generate an overall output tensor for computation.

[0142]

[0150] Although shown as separate components and circuits in Figure 8 for clarity, the decomposition circuit 826, multiplication circuit 827, inverse quantization circuit 828, and aggregation circuit 829 can be implemented collectively or individually within other processing devices of the processing system 800, such as within the CPU 802, GPU 804, DSP 806, and NPU 808.

[0143]

[0151] In general, the processing system 800 and / or its components may be configured to carry out the methods described herein.

[0144]

[0152] In particular, in other embodiments, elements of the processing system 800 may be omitted, such as when the processing system 800 is a server computer. For example, the multimedia component 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation processor 820 may be omitted in other embodiments. Furthermore, embodiments of the processing system 800 may be distributed among multiple devices.

[0145] Exemplary clause

[0153] The implementation forms are described in the following numbered clauses.

[0146]

[0154] Clause 1: A method comprising: accessing a parameter tensor of a machine learning model; identifying a set of rows in the parameter tensor, each containing one or more outliers; decomposing the parameter tensor into a first parameter subtensor corresponding to a set of rows and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor; quantizing the first parameter subtensor according to a first quantization scheme; quantizing the second parameter subtensor according to a second quantization scheme; and generating a quantized version of the machine learning model comprising the quantized first and second parameter subtensors.

[0147]

[0155] Clause 2: The method according to Clause 1, wherein the first quantization scheme includes a first quantized bit width, and the second quantization scheme includes a second quantized bit width smaller than the first quantized bit width.

[0148]

[0156] The method according to Clause 3: The method according to Clause 1 or 2, further comprising identifying one or more outliers, which includes determining one or more outlier criteria and evaluating each individual value in the parameter tensor using one or more outlier criteria.

[0149]

[0157] Clause 4: The method according to Clause 3, wherein one or more outlier criteria include a magnitude threshold.

[0150]

[0158] Clause 5: The method according to any one of Clauses 1 to 4, further comprising generating a set of outlier indices, which include the distinct indices of each distinct row in the set of rows, the set of outlier indices being used to decompose the input tensor during inference.

[0151]

[0159] Clause 6: The method according to Clause 5, wherein the inference includes accessing an input tensor, decomposing the input tensor into a first input subtensor having columns corresponding to a set of outlier indices, and a second input subtensor corresponding to at least one remaining column in the input tensor, generating a first output subtensor based on multiplying the first input subtensor by a first parameter subtensor, and generating a second output subtensor based on multiplying the second input subtensor by a second parameter subtensor.

[0152]

[0160] Clause 7: The method according to Clause 5 or 6, further comprising inferring the first and second output subtensors and generating an output tensor by summing the first and second inversely quantized output subtensors element by element.

[0153]

[0161] Clause 8: The method according to any of Clauses 1 to 7, wherein the parameter tensor includes weights for matrix multiplication operations.

[0154]

[0162] Clause 9: The method according to Clause 8, wherein the matrix multiplication operation is performed as part of at least one of the feedforward operation of the machine learning model or the attention operation of the machine learning model.

[0155]

[0163] Clause 10: A method comprising: accessing an input tensor of a layer of a machine learning model; decomposing the input tensor into a first input subtensor corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor; generating a first output subtensor based on multiplying the first input subtensor by a first parameter subtensor; generating a second output subtensor based on multiplying the second input subtensor by a second parameter subtensor; and generating an output tensor of a layer of a machine learning model based on the first and second output subtensors.

[0156]

[0164] Clause 11: The method according to Clause 10, wherein generating an output tensor includes dequantizing the first and second output subtensors and summing the first and second dequantized output subtensors element by element.

[0157]

[0165] Clause 12: The method according to Clause 10 or 11, wherein the first parameter subtensor is quantized according to the first quantization scheme and the second parameter subtensor is quantized according to the second quantization scheme.

[0158]

[0166] Clause 13: The method according to Clause 12, wherein the first quantization scheme includes a first quantized bit width, and the second quantization scheme includes a second quantized bit width that is greater than the first quantized bit width.

[0159]

[0167] Clause 14: The method of any of Clauses 10-13, which indicates a set of outlier indices that represents a set of rows in the unquantized parameter tensor of a layer of a machine learning model, each containing one or more outliers.

[0160]

[0168] Clause 15: The method according to any of Clauses 10 to 14, wherein the first and second parameter subtensors include weights for matrix multiplication operations.

[0161]

[0169] Clause 16: The method described in Clause 15, wherein the matrix multiplication operation is performed as part of at least one of the feedforward operations of the machine learning model or the attention operations of the machine learning model.

[0162]

[0170] Clause 17: A processing system comprising: a memory containing computer executable instructions; and one or more processors that execute computer executable instructions and are configured to cause the processing system to perform the methods described in any of Clauses 1 to 16.

[0163]

[0171] Clause 18: A processing system comprising means for carrying out the method described in any of Clauses 1 to 16.

[0164]

[0172] Clause 19: Non-temporary computer-readable media which, when executed by one or more processors of a processing system, includes computer-executable instructions causing the processing system to carry out any of the methods described in Clauses 1 to 16.

[0165]

[0173] Clause 20: A non-temporary computer-readable medium that, when executed by a processing system, encodes logic causing the processing system to carry out any of the methods described in Clauses 1 through 16.

[0166]

[0174] Clause 21: A device comprising a logic circuit configured to implement any of the methods described in Clauses 1 to 16.

[0167]

[0175] Clause 22: A computer program product embodied on a computer-readable storage medium, which includes code for carrying out any of the methods described in Clauses 1 through 16.

[0168] Additional considerations

[0176] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may also be applied to other embodiments. For example, changes may be made to the function and configuration of the elements discussed without departing from the scope of this disclosure. Various embodiments may omit, replace, or add various procedures or components as needed. For example, the methods described may be performed in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. Furthermore, the scope of this disclosure is intended to encompass apparatus or methods that are practiced using other structures, functions, or structures and functions in addition to, or other than, the various embodiments of this disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0169]

[0177] As used herein, the term “exemplary” means “serving as an example, illustration, or representation.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0170]

[0178] Where used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0171]

[0179] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0172]

[0180] The methods disclosed herein include one or more steps or actions to achieve those methods. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the claims. Furthermore, various operations of the methods described above can be performed by any preferred means capable of performing the corresponding functions. These means may include, but are not limited to, various hardware components and / or software components, and / or various hardware modules and / or software modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are shown in figures, those operations may have corresponding equivalent means-plus-function components with similar numbering.

[0173]

[0181] The following claims are not intended to be limited to the embodiments shown herein, but rather the full scope consistent with the language of the claims should be recognized. Within the claims, a singular reference to an element is intended to mean "one or more" rather than "one and only one" unless explicitly stated otherwise. Unless otherwise stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless it is explicitly enumerated using the phrase "means for..." or, in the case of a method claim, enumerated using the phrase "steps for...". All structural and functional equivalents of elements of various embodiments described throughout this disclosure, which are known to or will become known to those skilled in the art, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly enumerated in the claims.

Claims

1. A processing system, One or more memory locations containing processor-executable instructions, One or more processors that execute the processor-executable instructions and the processing system Allow access to the parameter tensor of the machine learning model, In the parameter tensor, a set of rows is identified, each containing one or more outliers. The parameter tensor is decomposed into a first parameter subtensor corresponding to the set of rows, and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor. The first parameter subtensor is quantized according to the first quantization scheme, The second parameter subtensor is quantized according to the second quantization scheme, To generate a quantized version of the machine learning model including the quantized first and second parameter subtensors, One or more processors configured as such, A processing system equipped with the following features.

2. The first quantization scheme includes a first quantized bit width, The second quantization scheme includes a second quantized bit width that is smaller than the first quantized bit width. The processing system according to claim 1.

3. The one or more processors are configured to further execute the processor-executable instructions in order to cause the processing system to identify the one or more outliers, and in order to identify the one or more outliers, the one or more processors are configured to cause the processing system to execute the processor-executable instructions Determine one or more outlier criteria, Use the one or more outlier criteria mentioned above to evaluate each individual value in the parameter tensor. The processing system according to claim 1, configured to execute the processor-executable instructions for this purpose.

4. The processing system according to claim 3, wherein the one or more outlier criteria include a magnitude threshold.

5. The processing system according to claim 1, wherein one or more processors are configured to further execute the processor executable instructions to cause the processing system to generate a set of outlier indices, each of which contains a separate index for each individual row in the set of rows, the set of outlier indices being used to decompose an input tensor during inference.

6. In order to perform inference, one or more second processors of the inference system provide the inference system with: Allow access to the input tensor, The input tensor is decomposed into a first input subtensor having columns corresponding to the set of outlier indices, and a second input subtensor corresponding to at least one remaining column in the input tensor. A first output subtensor is generated by multiplying the first input subtensor by the first parameter subtensor. The second output subtensor is generated by multiplying the second input subtensor by the second parameter subtensor. The processing system according to claim 5, configured to execute a second processor-executable instruction.

7. In order to perform inference, one or more second processors of the inference system perform the inference system The first and second output subtensors are dequantized, The output tensor is generated by summing the first and second inversely quantized output subtensors element by element. The processing system according to claim 6, configured to execute the second processor-executable instruction.

8. The processing system according to claim 1, wherein the parameter tensor includes weights for matrix multiplication operations.

9. The processing system according to claim 8, wherein the matrix multiplication operation is performed as part of at least one of the feedforward operation of the machine learning model or the attention operation of the machine learning model.

10. A processing system, One or more memory locations containing processor-executable instructions, One or more processors that execute the processor-executable instructions and the processing system Allow access to the input tensors of the layers in the machine learning model, The input tensor is decomposed into a first input subtensor corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor. The first output subtensor is generated by multiplying the first input subtensor by the first parameter subtensor. The second output subtensor is generated by multiplying the second input subtensor by the second parameter subtensor. Based on the first and second output subtensors, the output tensor of the layer of the machine learning model is generated. One or more processors configured as such, A processing system equipped with the following features.

11. In order to generate the output tensor, one or more processors provide the processing system with The first and second output subtensors are dequantized, The first and second inversely quantized output subtensors are summed element by element. The processing system according to claim 10, further configured to execute the processor-executable instructions.

12. The first parameter subtensor is quantized according to the first quantization scheme, The aforementioned second parameter subtensor is quantized according to the second quantization scheme. The processing system according to claim 10.

13. The first quantization scheme includes a first quantized bit width, The second quantization scheme includes a second quantized bit width that is larger than the first quantized bit width. The processing system according to claim 12.

14. The processing system according to claim 10, wherein the set of outlier indices represents a set of rows in the unquantized parameter tensor of the layer of the machine learning model, each of which contains one or more outliers.

15. The processing system according to claim 10, wherein the first and second parameter subtensors include weights for matrix multiplication operations.

16. The processing system according to claim 15, wherein the matrix multiplication operation is performed as part of at least one of the feedforward operation of the machine learning model or the attention operation of the machine learning model.

17. A method implemented by the processor, Accessing the parameter tensor of a machine learning model, Identifying a set of rows in the parameter tensor, each containing one or more outliers, Decomposing the parameter tensor into a first parameter subtensor corresponding to the set of rows and a second parameter subtensor corresponding to at least one remaining row in the parameter tensor, Quantizing the first parameter subtensor according to the first quantization scheme, Quantizing the second parameter subtensor according to the second quantization scheme, To generate a quantized version of the machine learning model that includes the quantized first and second parameter subtensors, A method implemented by the processor, including the methods mentioned above.

18. The first quantization scheme includes a first quantized bit width, The second quantization scheme includes a second quantized bit width that is smaller than the first quantized bit width. A method implemented by the processor described in claim 17.

19. Determining one or more outlier criteria, Evaluating each individual value in the parameter tensor using the one or more outlier criteria mentioned above, A method implemented by the processor according to claim 17, further comprising identifying one or more outliers, including the above.

20. A method by which the processor according to claim 19 implements the one or more outlier criteria, including a magnitude threshold.

21. A method implemented by the processor according to claim 17, further comprising generating a set of outlier indices, each of which has a distinct index in the set of rows, wherein the set of outlier indices is used to decompose an input tensor during inference.

22. The above reasoning is, Accessing the input tensor, Decomposing the input tensor into a first input subtensor having columns corresponding to the set of outlier indices, and a second input subtensor corresponding to at least one remaining column in the input tensor, The first output subtensor is generated by multiplying the first input subtensor by the first parameter subtensor, The second output subtensor is generated by multiplying the second input subtensor by the second parameter subtensor, A method implemented by the processor according to claim 21, including the above.

23. The above reasoning is, The first and second output subtensors are dequantized, The output tensor is generated by summing the first and second inversely quantized output subtensors element by element, A method implemented by the processor according to claim 22, further comprising:

24. The method implemented by the processor according to claim 17, wherein the parameter tensor includes weights for matrix multiplication operations.

25. The method implemented by the processor according to claim 24, wherein the matrix multiplication operation is implemented as part of at least one of the feedforward operation of the machine learning model or the attention operation of the machine learning model.

26. A method implemented by the processor, Accessing the input tensors of the layers in a machine learning model, Decomposing the input tensor into a first input subtensor corresponding to a set of outlier indices and a second input subtensor corresponding to at least one remaining element in the input tensor, The first output subtensor is generated by multiplying the first input subtensor by the first parameter subtensor, The second output subtensor is generated by multiplying the second input subtensor by the second parameter subtensor, The output tensor of the layer of the machine learning model is generated based on the first and second output subtensors, A method implemented by the processor, including the methods mentioned above.

27. To generate the aforementioned output tensor, The first and second output subtensors are dequantized, The first and second inversely quantized output subtensors are summed element by element, A method implemented by the processor according to claim 26, including the above.

28. The first parameter subtensor is quantized according to the first quantization scheme, The aforementioned second parameter subtensor is quantized according to the second quantization scheme. A method implemented by the processor described in claim 26.

29. The first quantization scheme includes a first quantized bit width, The second quantization scheme includes a second quantized bit width that is larger than the first quantized bit width. A method implemented by the processor described in claim 28.

30. A method implemented by the processor according to claim 26, wherein the set of outlier indices represents a set of rows in the unquantized parameter tensor of the layer of the machine learning model, each of which contains one or more outliers.

31. A method implemented by the processor according to claim 26, wherein the first and second parameter subtensors include weights for matrix multiplication operations.

32. The method implemented by the processor according to claim 31, wherein the matrix multiplication operation is implemented as part of at least one of the feedforward operation of the machine learning model or the attention operation of the machine learning model.

Images