Quantization compensation for machine learning models

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. A first machine learning model is accessed, the first machine learning model comprising a first plurality of blocks associated with a first precision. A second machine learning model comprises a second plurality of blocks associated with a second precision, wherein the second plurality of blocks includes a first block that corresponds to a first block of the first plurality of blocks. Processing an input to the first machine learning model using the first plurality of blocks and the second plurality of blocks includes modifying an output of the first block of the first plurality of blocks based on the corresponding first block of the second plurality of blocks. An output of the first machine learning model is provided based on the processing.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 514,602, filed November 20, 2023, which is incorporated herein by reference.

[0003] introduction

[0004] This disclosure relates to various aspects of machine learning.

[0005] Recently, a wide variety of machine learning architectures have been used to perform countless tasks with high accuracy and reliability. For example, computer vision models have been used to perform tasks such as object detection and distance prediction. Language models (e.g., Large Language Models (LLMs)) have been used to understand and generate text output in a human-like manner, such as for chatbots. However, many existing model architectures are large (e.g., with thousands, millions, or billions of parameters), and training such models typically relies on massive amounts of training data (and comes with equally enormous computational overhead).

[0006] Some common methods used to improve accessibility to machine learning (e.g., on edge devices with limited computing power) include model quantization. While quantization can significantly reduce the size of models, it also introduces inherent errors because high-precision model parameters are approximated using lower-precision values. Summary of the Invention

[0007] Some aspects of this disclosure provide a processor-implemented method comprising: accessing a first machine learning model, the first machine learning model including a first plurality of blocks, the first plurality of blocks being associated with and including a first precision; accessing a second machine learning model, the second machine learning model including a second plurality of blocks, the second plurality of blocks being associated with a second precision different from the first precision, wherein: the second plurality of blocks include the first block; and the first block in the second plurality of blocks corresponds to the first block in the first plurality of blocks; processing an input to the first machine learning model using the first plurality of blocks of the first machine learning model and the second plurality of blocks of the second machine learning model, wherein the processing includes modifying the output of the first block in the first plurality of blocks based on the corresponding first block in the second plurality of blocks; and providing an output of the first machine learning model based on the processing.

[0008] Some aspects of this disclosure provide a processor-implemented method comprising: accessing a first machine learning model including a first plurality of blocks; generating a second machine learning model including a second plurality of blocks by quantizing the baseline machine learning model; training a third machine learning model for adjusting the quantization of the first machine learning model, the third machine learning model including a third plurality of blocks; and deploying the second machine learning model and the third machine learning model for inference.

[0009] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0010] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0011] The accompanying drawings illustrate certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0012] Figure 1 An example workflow for compensating and adapting a quantized machine learning model, based on some aspects of this disclosure, is described.

[0013] Figure 2A , Figure 2B and Figure 2C An example workflow for generating output using a machine learning model with quantized compensation, according to some aspects of this disclosure, is described.

[0014] Figure 3 This is a flowchart depicting an example method for compensating and adapting a quantized machine learning model according to some aspects of this disclosure.

[0015] Figure 4 This is a flowchart depicting an example method for training a quantization compensation model according to some aspects of this disclosure.

[0016] Figure 5 This is a flowchart depicting an example method for adapting a quantization compensation model according to some aspects of this disclosure.

[0017] Figure 6 This is a flowchart depicting an example method for generating output using a machine learning model with quantized compensation, according to some aspects of this disclosure.

[0018] Figure 7 This is a flowchart depicting an example method for using a machine learning model to compensate for quantization, according to some aspects of this disclosure.

[0019] Figure 8 This is a flowchart depicting an example method for training a machine learning model to compensate for quantization, according to some aspects of this disclosure.

[0020] Figure 9 Example processing systems configured to perform various aspects of this disclosure are described.

[0021] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0022] This disclosure provides apparatus, methods, processing systems, and non-transitory computer-readable media for providing improved machine learning.

[0023] In some respects, a baseline machine learning model (e.g., an LLM) can be quantized using one or more quantization operations to obtain a quantized version of the model, where the quantized version is relatively small in terms of memory usage (e.g., the parameters can be stored using relatively fewer bits compared to the unquantized baseline model). Typically, to quantize a model, fewer bits (e.g., eight-bit or four-bit representation) are used to approximate one or more of the model parameters (which are typically represented in a high-precision format such as 16-bit floating-point). Since the resulting quantized weights are stored using fewer bits, the model size can be significantly reduced. However, such quantization inherently introduces quantization error, where the output of the quantized model is often less accurate or reliable than the output of the unquantized model. Generally, a more aggressive quantization scheme (e.g., quantizing to four bits instead of eight bits) results in a smaller size but an increased error.

[0024] In some aspects of this disclosure, a quantized baseline model can have its weights frozen and can be trained with one or more small, high-precision compensation modules to compensate for (or at least reduce) quantization errors. For example, the compensation modules can be less than one percent the size of the quantized model. Typically, small compensation blocks are much easier to train (e.g., involving fewer computational resources and fewer examples) compared to much larger baseline models. However, such compensation modules can significantly improve model accuracy. For example, a model quantized to four bits can be augmented with a set of small sixteen-bit compensation modules, and the resulting combination can produce outputs with accuracy comparable to models with higher quantization precision (e.g., an eight-bit quantized model without compensation modules, which is approximately twice the size of a four-bit quantized model).

[0025] Typically, the size of a machine learning model is the product of the number of parameters and the precision (e.g., bit width) used to store those parameters. Using aspects of this disclosure, a model with a relatively large number of parameters encoded with relatively low precision can be coupled with a compensating model with a relatively small number of parameters encoded with relatively high precision. This combined model can provide comparable (or higher) accuracy to a model with a large number of parameters encoded with high precision while using less memory space and computational resources.

[0026] In some respects, in addition to training the compensation module to compensate for (or at least reduce) quantization errors, task-specific data can be used to train the compensation module (while keeping the quantized baseline model frozen). This efficient task adaptation can be similarly performed with minimal resources (e.g., on edge devices) and can significantly improve model output. For example, combining a four-bit quantized machine learning model with a sixteen-bit compensation model trained using task data can produce a more accurate output than the original (unquantized) model itself, which may be much larger (e.g., in sixteen-bit floating-point format, approximately four times the size of the original).

[0027] In some aspects, depending on a specific implementation, the quantized machine learning model and the compensation model can be executed or processed using different hardware components. For example, the quantized model can be deployed (e.g., processed) using a first component (e.g., a first integrated circuit (IC) device) that is relatively more suitable for low-precision (e.g., four-bit) representations and / or a large number of parameters (e.g., for a component that can process a large number of parameters in parallel), while the compensation model can be deployed to a second hardware component (e.g., a second IC device) that is relatively more suitable for high-precision (e.g., 16-bit) representations and / or a smaller number of parameters. In this way, the combined model (including the quantized model and the compensation model) can be executed efficiently. In some aspects, the parts of the model can be executed in parallel. For example, depending on a specific implementation, each module of the quantized model can be executed substantially in parallel with the first component by a hardware component, while the corresponding block of the compensation model is processed by the second hardware component.

[0028] In some respects, the compensation model can be trained block-wise (e.g., on a layer-by-layer basis) and / or end-to-end. As used in this paper, a “block” of a machine learning model typically corresponds to a logical segment of the model, such as a layer (of a neural network), a transformer, an MLP, etc. In some respects, for each block of the quantized model, a corresponding compensation block of the compensation model can be trained. In some respects, block-wise training of the compensation model can typically include attempting to minimize the loss for intermediate features generated by the corresponding block, while end-to-end training can typically include attempting to minimize the loss for the overall model output.

[0029] In some respects, the training of the compensation model is performed using the formula defined in Equation 1 below, where It is a compensation model The parameters, It is the output of the original (unquantized) machine learning model given the input. (or the output of a given block of the original model). It is the output of the quantified machine learning model given the input. (or the output of the quantized block corresponding to the quantized model of the given block), and It is the output of the compensation model (or the compensation block corresponding to a given block of the original model).

[0030] (1)

[0031] Advantageously, since the compensated model typically has fewer parameters compared to the baseline model, it can be efficiently trained on a wider variety of devices (e.g., resource-constrained devices such as, but not limited to, personal smartphones, tablets, wearables, Internet of Things (IoT) devices, etc.). Furthermore, fewer training examples are needed to train the smaller compensated model. This allows end-users to perform efficient compensation and adaptation. Furthermore, training on such devices can enhance or maintain user privacy (e.g., because personal data does not need to be provided to more powerful remote and / or cloud-based training servers). Additionally, in some respects, since the baseline model and the quantized model are frozen during the training of the compensated model, there is no need to store the optimized state of those large models that are typically saved and used throughout the training cycle. This further reduces the resources required to train the compensated model. Additionally, since the compensated model can be implemented using conventional high-precision (e.g., 16-bit floating-point) parameters, standard methods can be used to train the compensated model, avoiding the additional operations commonly used to learn the quantized weights.

[0032] Example workflow for compensating and adapting the quantized machine learning model

[0033] Figure 1 An example workflow 100 for compensating and adapting a quantized machine learning model according to some aspects of this disclosure is described. In some aspects, workflow 100 is implemented by one or more processing systems, such as a machine learning system that trains the machine learning model, a machine learning system that performs inference using the trained model (e.g., an edge device), etc.

[0034] In some aspects, the operations described in workflow 100 and discussed in more detail below can be distributed across multiple devices and systems. That is, the training component 110, quantization component 120, compensation component 135, and adaptation component 150 (which may each be implemented using hardware, software, or a combination of hardware and software) can be components of a single processing system or can be distributed across multiple processing systems. For example, in some aspects, the training component 110, quantization component 120, and / or compensation component 135 can be implemented by a server that performs model training, while the adaptation component 150 can be implemented by a device that will use the trained model during inference.

[0035] In the illustrated workflow 100, the training component 110 accesses a set of training data 105. As used herein, “accessing” data can generally include receiving, requesting, capturing, generating, collecting, obtaining, or otherwise acquiring access to the data. Although depicted as a discrete repository of training data 105 for clarity of concept, in some respects, training data 105 may be distributed across any number of repositories or other data sources. The specific format and content of training data 105 may vary depending on the specific task to which the machine learning model is being trained. For example, for a computer vision task, training data 105 may include images and corresponding labels (e.g., segmentation maps, classifications, etc.). As another example, if an LLM is being trained, training data 105 may include text samples.

[0036] As illustrated, training component 110 uses training data 105 to generate (e.g., train) machine learning model 115. The specific architecture of machine learning model 115 may vary depending on the specific implementation and may include architectures such as LLM, CNN, transformer-based models, etc. In some respects, as discussed above, machine learning model 115 may be referred to as a baseline model. The parameters of machine learning model 115 are typically represented or encoded using high-precision value representations (such as hexadecimal floating-point numbers).

[0037] In the illustrated workflow 100, machine learning model 115 is accessed by quantization component 120. Quantization component 120 is typically configured to quantize the parameters of machine learning model 115 into a lower bit-width representation, thereby reducing the size of the model. As illustrated, this results in a quantized model 125. For example, the parameters of the quantized model 125 may be represented or encoded using lower-precision value representations, such as four-bit or eight-bit integers. In some aspects, the quantized model 125 may be referred to as the “quantized machine learning model” and / or “quantized version” of the baseline machine learning model. In some aspects, quantization may cause at least some of the parameters of the quantized model 125 to take a value of zero. Therefore, in some aspects, quantization component 120 may remove such zero-value parameters from the quantized model 125, thereby further reducing the size of the quantized model and reducing the number of parameters.

[0038] As illustrated, the subsequently quantized model 125 is accessed by the compensation component 135. The compensation component 135 generates (e.g., trains) a compensated model 140 using a set of compensation data 130. In some aspects, the compensation data 130 may alternatively be referred to as adjustment data. In some aspects, the compensated model 140 is a whole comprising the quantized model 125 and the separate compensation model. In some aspects, as discussed above, the parameters of the compensation model may be represented or encoded using higher precision values ​​(such as hexadecimal floating-point). However, since the parameters of the compensation model may be much fewer than those of the quantized model 125 (e.g., less than 1%), the increase in the overall size of the compensated model 140 relative to the quantized model 125 is negligible.

[0039] In some respects, the compensation data 130 typically includes data from the same task as the training data 105. For example, if the training data 105 includes image input, the compensation data 130 may similarly include images. In some respects, the compensation data 130 does not need to include labels. That is, while the samples in the training data 105 may have corresponding ground truth labels to facilitate the training of the machine learning model 115, the compensation data 130 may not have such labels. For example, since the compensation model is trained to minimize (or at least reduce) the difference between the quantified model 125 and the machine learning model 115, the actual ground truth output is irrelevant during training.

[0040] In some aspects, to train the compensation model, the compensation component 135 can generate a first output by processing samples from the compensation data 130 using a machine learning model 115, a second output by processing samples using a quantized model 125, and a third output by processing samples using the compensation model. The compensation component 135 can then compute a loss between the first output and the aggregation (e.g., sum) of the second and third outputs. This loss can be used to refine the parameters of the compensation model (e.g., using backpropagation). As discussed above, this training can be performed block-by-block (e.g., computed with respect to intermediate features generated by each block) and / or end-to-end (e.g., computed with respect to the final model output). Similarly, this training can be performed using individual samples (e.g., stochastic gradient descent) and / or batch samples (e.g., batch gradient descent).

[0041] In some respects, the compensated model 140 may then be deployed or provided to an inference system for use. For example, the compensated model 140 may be generated by a first system (e.g., a server that performs model training and quantization) and deployed to a second system (e.g., a user device such as a laptop computer or smartphone). In some respects, the compensated model 140 may be accessed by an adaptation component 150 (e.g., on a training server or on an edge device) rather than for inference.

[0042] As illustrated, adaptation component 150 accesses adaptation data 145 and generates (e.g., trains) an adapted model 155. In some aspects, adaptation data 145 typically includes data from the same task as training data 105 (e.g., images), but this data is geared towards the target domain (where training data 105 may come from the source domain). For example, adaptation data 145 may include image examples specific to the user or system that will use the adapted model 155 (while training data 105 may be agnostic or generalizable to a large number of users). Such task-specific data enables efficient task adaptation, thereby significantly improving predictions by using personalized models.

[0043] In some respects, the adaptation data 145 includes ground truth labels in a manner similar to that of the training data 105. In some respects, in order to train the adapted model 155, some parameters of the compensated model 140 (e.g., the parameters of the quantized model 125) may be frozen or static (e.g., not changed), while other parameters (e.g., the parameters of the compensated model) may be updated.

[0044] In some aspects, to train the adapted model 155, the adaptation component 150 can generate output by processing samples from the adapted data 145 using the compensated model 140. The adaptation component 150 can then compute a loss between the output and the ground truth labels of the samples. This loss can be used to refine the parameters of the compensated portion of the compensated model 140 (e.g., using backpropagation). As discussed above, this training can be performed block-by-block (e.g., computed loss with respect to intermediate features generated by each block) and / or end-to-end (e.g., computed loss with respect to the final model output). Similarly, this training can be performed using individual samples (e.g., stochastic gradient descent) and / or batch samples (e.g., batch gradient descent). In some aspects, the quantized portion of the compensated model 140 is frozen during adaptation.

[0045] As illustrated, the adapted model 155 may then be deployed or otherwise provided for inference. For example, the adapted model 155 may be deployed or implemented on one or more hardware components and used to process input data to generate output during runtime.

[0046] Advantageously, the adapted model 155 is typically much smaller than the machine learning model 115 (in terms of size or memory usage), but its accuracy is comparable (or even improved).

[0047] Example workflow for generating output using a machine learning model with quantized compensation.

[0048] Figure 2A , Figure 2B and Figure 2CExample workflows 200A to 200C are described according to some aspects of this disclosure for generating output using a machine learning model with quantified compensation. In some aspects, workflows 200A to 200C are performed by one or more processing systems (such as those referenced above). Figure 1 The machine learning system discussed is implemented in this way.

[0049] Specifically, Figure 2A The workflow 200A describes an architecture in which each block of the compensation model processes intermediate features as part of the computation of each block of the quantized model. Figure 2B The workflow 200B describes an architecture in which each block of the compensation model operates in parallel with the corresponding block in the quantized model to generate inputs for subsequent blocks, and Figure 2C Workflow 200C describes an architecture in which the output of each block of the quantified model is transformed or updated by the corresponding block of the compensation model. Generally, workflows 200A to 200C (collectively referred to as workflow 200) can represent alternative or various specific implementations of the overall compensated model, and each workflow can be used in a given model. That is, in some respects, the elements of each architecture can be combined to form a single compensation model.

[0050] like Figure 2A The model described in the text, which compensates for the whole (which can correspond to...) Figure 1 The compensated model 140 includes a quantized model 125 and a compensated model 210. The quantized model 125 includes a set of blocks 215A to 215E (collectively referred to as blocks 215), while the compensated model 210 includes a set of corresponding blocks 220A to 220E (collectively referred to as blocks 220). In some aspects, the quantized model 125 includes an ordered set or sequence of blocks 215, which are processed sequentially.

[0051] Typically, the specific operations performed by each block 215 and 220 may vary depending on the specific implementation. For example, each block 215 and 220 may perform one or more convolution operations, attention operations, transformer operations, etc. In some aspects, each block in block 220 of the compensation model 210 includes a multilayer perceptron (MLP).

[0052] In the illustrated workflow 200A, input 205 is accessed by the first block 215A of the quantized model 125. As illustrated, block 215A performs one or more operations or transformations (e.g., convolution) on input 205 to generate a first intermediate feature set. These features are then fed to a first block 220A (which corresponds to block 215A), which performs one or more operations (e.g., convolution) to generate a second intermediate feature set. This second intermediate feature set is then processed by block 215A to generate an output intermediate feature set from block 215A. This output intermediate feature set is then used as input to block 215B.

[0053] As illustrated, each block 215 of the quantized model 125 has a corresponding block 220 in the compensated model 210, wherein the intermediate features of each block 215 are processed or transformed by the corresponding block 220, and the output of each block 215 is generated at least in part based on the (compensated) output of the corresponding block 220. As shown, block 215E generates output 225 from the compensated model. As discussed above, the specific format and content of input 205 and output 225 may vary depending on the specific implementation. For example, if the compensated model is an LLM architecture, both input 205 and output 225 may include natural language text.

[0054] Although five blocks 215 and 220 are depicted for conceptual clarity, any number of blocks 215 and 220 may exist in the quantified model 125 and the compensation model 210. Furthermore, while the illustrated example depicts a corresponding block 220 for each block 215, in some respects, the compensation model 210 may include only a subset of blocks 220 for blocks 215. For example, there may be compensation blocks 220 for one or more earlier blocks 215 (e.g., for the first block 215A) and for one or more later blocks 215 (e.g., for the last block 215E), but one or more inner blocks 215 (e.g., blocks 215B, 215C, and 215D) may lack corresponding compensation blocks 220.

[0055] In some aspects, for the architecture using workflow 200A, the training system may train the compensation model 210 end-to-end and / or block-by-block. In some aspects, for end-to-end training of the compensation model 210, the training system may update the parameters of block 220 (keeping the parameters of block 215 unchanged) to minimize (or at least reduce) the difference between output 225 and the corresponding output generated when input 205 is processed by the unquantized baseline model. Similarly, for block-by-block training of the compensation model 210, the training system may update the parameters of block 220 to minimize (or at least reduce) the difference between the output of each block 215 and the output of the corresponding block in the unquantized baseline model.

[0056] Go to Figure 2B The overall compensation model (which can correspond to) Figure 1 The compensated model 140 similarly includes the quantized model 125 and the compensation model 210, wherein the quantized model 125 includes a set of blocks 215A to 215E (collectively referred to as block 215), and the compensation model 210 includes a set of corresponding blocks 220A to 220E (collectively referred to as block 220). As discussed above, the specific operations performed by each block 215 and 220 may vary depending on the specific implementation.

[0057] In the illustrated workflow 200B, input 205 is accessed by the first block 215A of the quantized model 125. As shown, block 215A performs one or more operations or transformations (e.g., convolution) on input 205 to generate an output set of intermediate features. Furthermore, as illustrated, input 205 is also accessed by the first block 220A of the compensation model 210. The compensation model 210 similarly performs one or more operations (e.g., convolution) to generate a second set of output features. In some respects, as discussed above, blocks 215A and 220A can be performed or processed in parallel (e.g., on different hardware components).

[0058] As illustrated, the outputs of block 215A and block 220A are then combined via a corresponding aggregation operation 230A. Aggregation operation 230A can typically perform any number of operations (such as element-wise summation, element-wise averaging, element-wise maximization or minimization, etc.). Although depicted as a component of the quantized model 125, in some respects, aggregation operation 230A can be performed as a standalone operation of the compensated model (e.g., not directly as part of either the quantized model 125 or the compensated model 210).

[0059] In the illustrated example, the output of subsequent aggregation operation 230A is accessed by both block 215B of the quantized model 125 and block 220B of the compensated model 210. The resulting output is then aggregated again via the corresponding aggregation operation 230B, and this sequence is repeated for each block. As illustrated, output 225 is generated by aggregating the final outputs of blocks 215E and 220E using the corresponding aggregation operation 230E.

[0060] In the illustrated example, each block 215 of the quantized model 125 has a corresponding block 220 in the compensation model 210, wherein the intermediate features of each block 215 are aggregated with the features generated by the corresponding block 220 (via the corresponding aggregation operation 230), and the aggregated output is then provided as input to the next block (e.g., the next block 215 of the quantized model and the next block 220 of the compensation model).

[0061] As discussed above, although five blocks 215 and 220 are depicted for conceptual clarity, any number of blocks 215 and 220 may exist in the quantized model 125 and the compensation model 210. Furthermore, while the illustrated example depicts a corresponding block 220 for each block 215, in some respects, the compensation model 210 may only include blocks 220 for a subset of blocks 215, as discussed above. For example, if there is no block 220B in the compensation model 210 (e.g., block 215B of the quantized model 125 does not have a corresponding block in the compensation model 210), the output of block 215B could instead be directly provided to block 215C.

[0062] In some aspects, for the architecture using workflow 200B, the training system may train the compensation model 210 end-to-end and / or block-by-block. In some aspects, for end-to-end training of the compensation model 210, the training system may update the parameters of block 220 (while keeping the parameters of block 215 unchanged) to minimize (or at least reduce) the difference between output 225 and the corresponding output generated when input 205 is processed by the unquantized baseline model, as discussed above. Similarly, for block-by-block training of the compensation model 210, the training system may update the parameters of block 220 to minimize (or at least reduce) the difference between the output of each aggregation operation 230 and the output of the corresponding block in the unquantized baseline model.

[0063] Now transferred to Figure 2C The overall compensation model (which can correspond to) Figure 1 The compensated model 140 similarly includes the quantized model 125 and the compensation model 210, wherein the quantized model 125 includes a set of blocks 215A to 215E (collectively referred to as block 215), and the compensation model 210 includes a set of corresponding blocks 220A to 220E (collectively referred to as block 220). As discussed above, the specific operations performed by each block 215 and 220 may vary depending on the specific implementation.

[0064] In the illustrated workflow 200C, input 205 is accessed by the first block 215A of the quantized model 125, which generates an output set of features. These features are used as input to the corresponding block 220A of the compensated model 210. Block 220A then processes (e.g., transforms) the features using one or more operations. The transformed (e.g., compensated) features can then be provided as input to subsequent blocks 215B of the quantized model 125. This process can be repeated for each block until output 225 is generated by the final block 220E of the compensated model 210.

[0065] In the illustrated example, each block 215 of the quantized model 125 has a corresponding block 220 in the compensation model 210, where intermediate features of each block 215 are used as input to the corresponding block 220, and the output of the corresponding block 220 is subsequently provided as input to the next block 215 of the quantized model. As discussed above, although five blocks 215 and 220 are depicted for conceptual clarity, any number of blocks 215 and 220 may exist in the quantized model 125 and the compensation model 210. Furthermore, although the illustrated example depicts a corresponding block 220 for each block 215, in some respects, the compensation model 210 may only include blocks 220 for a subset of blocks 215, as discussed above. For example, if there is no block 220B in the compensation model 210 (e.g., block 215B of the quantized model 125 does not have a corresponding block in the compensation model 210), the output of block 215B may instead be provided directly to block 215C.

[0066] In some aspects, for the architecture using workflow 200C, the training system may train the compensation model 210 end-to-end and / or block-by-block. In some aspects, for end-to-end training of the compensation model 210, the training system may update the parameters of block 220 (keeping the parameters of block 215 unchanged) to minimize (or at least reduce) the difference between output 225 and the corresponding output generated when input 205 is processed by the unquantized baseline model, as discussed above. Similarly, for block-by-block training of the compensation model 210, the training system may update the parameters of block 220 to minimize (or at least reduce) the difference between the output of each block 220 and the output of the corresponding block in the unquantized baseline model.

[0067] In some respects, as discussed above, the architectures discussed in the reference workflows 200A, 200B, and 200C above can be combined within a single compensated model. For example, one or more quantized blocks 215 of the quantized model 125 can use corresponding compensation blocks 220 of the compensated model 210 to handle individual internal or intermediate features (such as reference models). Figure 2A (As discussed), the outputs of one or more quantized blocks 215 of the quantized model 125 can be aggregated with the output features of the corresponding compensation block 220 of the compensation model 210 (as referenced). Figure 2B (as discussed), and / or one or more quantized blocks 215 of the quantized model 125 may receive input from the compensation block 220 and provide the output to the corresponding compensation block 220 of the compensation model 210 (as referenced). Figure 2C(As discussed). Similarly, one or more quantized blocks 215 of the quantized model 125 may not have a corresponding compensation block 220 (e.g., the output of block 215 may be provided as input to subsequent blocks 215), and / or one or more additional compensation blocks 220 may be used (e.g., to generate model output in the absence of a corresponding quantized block 215).

[0068] Example methods for compensating and adapting the quantized machine learning model

[0069] Figure 3 This is a flowchart depicting an example method 300 for compensating and adapting a quantized machine learning model according to some aspects of this disclosure. In some aspects, method 300 comprises one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B and / or Figure 2C The machine learning system under discussion is executed.

[0070] At box 305, the system accesses the trained machine learning model (e.g., Figure 1 The machine learning model 115). In some respects, as discussed above, the processing system may use training data to train the machine learning model. In other respects, the processing system may access or receive a trained machine learning model from one or more other systems. The trained machine learning model typically includes a set of parameters whose values ​​are learned based on the training data. For example, the machine learning model may correspond to a convolutional neural network (CNN) (e.g., for computer vision tasks), an LLM (e.g., for text generation tasks), etc. In some respects, the machine learning model is a relatively large model (e.g., with a large number of parameters encoded using a relatively high-precision format, such as hexadecimal).

[0071] At box 310, the processing system generates a quantized machine learning model by quantizing the trained machine learning model (e.g., Figure 1 and Figures 2A to 2C The quantized model (125). In some respects, as discussed above, the quantized machine learning model can typically have a relatively small size compared to the trained machine learning model. For example, the parameters of the quantized machine learning model can be encoded using relatively low-precision formats (such as four-bit or eight-bit).

[0072] At box 315, the system training quantization compensation model is processed (e.g., ...). Figures 2A to 2CThe compensation model 210). In some aspects, as discussed above, the processing system trains the quantization compensation model based on the trained machine learning model and the quantized machine learning model, aiming to make the output of the quantized model similar to the output of the trained model. In some aspects, as discussed above, the processing system uses unlabeled training data (e.g., Figure 1 The compensation data (130) is used to update the parameters of the compensation model, while the parameters of the trained machine learning model and the quantized machine learning model remain fixed. In some aspects, as discussed above, the processing system trains the compensation model end-to-end. In other aspects, as discussed above, the processing system trains the compensation model in a block-by-block manner.

[0073] At box 320, the processing system may optionally adapt the quantization compensation model to generate the adapted model (e.g., Figure 1 The model adapted to it (155), as discussed above. For example, as discussed above, the processing system can use labeled training data for the target domain (e.g., Figure 1 The adapted data (145) is used to update the parameters of the quantized compensation model, while the parameters of the quantized machine learning model remain fixed. In some respects, as discussed above, the processing system adapts the compensation model end-to-end. In other respects, as discussed above, the processing system trains the compensation model in a block-by-block manner.

[0074] At box 325, the processing system deploys the quantized machine learning model and (potentially adapted) quantization compensation model (e.g., the overall quantized and compensation models) for inference. In some respects, as discussed above, different components of the compensation model can be implemented or deployed using different hardware components. For example, the quantized machine learning model can be processed using one hardware component (e.g., a graphics processing unit (GPU)), while the (potentially adapted) compensation model can be processed using a second hardware component (e.g., a central processing unit (CPU)).

[0075] Example methods for training quantization compensation models

[0076] Figure 4 This is a flowchart depicting an example method 400 for training a quantization compensation model according to some aspects of this disclosure. In some aspects, method 400 comprises one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B , Figure 2C and / or Figure 3 The machine learning system under discussion is executed. In some respects, method 400 is aimed at... Figure 3 Box 315 provides additional details.

[0077] At box 405, the processing system selects a compensation sample (e.g., from...). Figure 1 The compensation data consists of 130 samples. Typically, the processing system can use any suitable criteria (including random or pseudo-random) to select the compensation samples. In some respects, as discussed above, the compensation samples may often correspond to the same domain as the data used to train the baseline model, but may lack ground truth labels.

[0078] At box 410, the processing system uses a block of the quantized model (e.g., Figures 2A to 2C The system uses blocks 215 of the quantized model 125 to generate a first feature map. In some aspects, the specific data processed to generate the feature map may vary depending on the specific implementation. For example, if the block is the first block in the quantized model, the processing system can use that block to process selected compensation samples to generate the first feature map. If the block is an inner block of the quantized model, the processing system can process the feature map generated by previous components.

[0079] For example, if the compensated model uses a reference Figure 2A In the architecture discussed, the processing system can use the current block to process the feature maps generated by previous blocks of the quantized model in order to generate a first feature map. For example, if the compensated model uses a reference... Figure 2B In the architecture discussed, the processing system can use the current block to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate the first feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use the current block of the quantized model to process the feature map generated by the previous block of the compensated model in order to generate the first feature map.

[0080] At box 415, the processing system determines whether there is a corresponding compensation block for the quantized block used to generate the first feature map at box 410. If not, method 400 proceeds to box 435. If there is a corresponding compensation block in the compensation model, method 400 continues to box 420.

[0081] At box 420, the processing system uses the corresponding block of the compensation model (e.g., Figures 2A to 2C The compensation model 210 (block 220) is used to generate the second feature map. In some respects, the specific data processed to generate the second feature map may vary depending on the specific implementation.

[0082] For example, if the compensated model uses a reference Figure 2A The architecture discussed allows the processing system to handle intermediate feature maps generated from corresponding blocks of the quantized model, as discussed above. For example, if the compensated model uses a reference... Figure 2BIn the architecture discussed, the processing system can use compensation blocks to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate a second feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use compensation blocks to process the feature maps generated by the corresponding blocks of the quantized model (e.g., at box 410) in order to generate a second feature map.

[0083] At box 425, the subsequent processing system optionally computes a block-by-block compensation loss for the compensation model based on the first and second feature maps (generated at boxes 410 and 425, respectively). In some respects, as discussed above, the generation of the compensation loss is further based on feature maps generated from corresponding blocks in the unquantized baseline model. For example, the processing system may compute a cross-entropy loss based on the features (or use any other suitable loss algorithm). Method 400 then continues to box 435.

[0084] At box 435, the processing system determines whether there is at least one more block in the quantized model. If so, method 400 returns to box 410 to generate a new feature map using the next block in the quantized model. If no more blocks remain, method 400 continues to box 440.

[0085] At box 440, the processing system optionally generates an end-to-end compensation loss based on the output of the compensated model (e.g., the final output of the quantized model and / or the compensated model). In some respects, as discussed above, the generation of the compensation loss is further based on the output generated from the unquantized baseline model. For example, the processing system may compute a cross-entropy loss based on the output (or use any other suitable loss algorithm). Method 400 then continues to box 445.

[0086] At box 445, the processing system updates one or more parameters of the compensation model (e.g., using backpropagation), as discussed above. In some respects, the parameters of the quantized machine learning model (and the baseline model) are static and invariant during the updates of the compensation model. While the illustrated example depicts stochastic gradient descent (e.g., refining the compensation model individually based on each compensation example) for conceptual clarity, in some respects, the processing system may additionally or alternatively use batch gradient descent.

[0087] At box 450, the processing system determines whether one or more termination criteria are met. Typically, the termination criteria used may vary depending on the specific implementation. For example, in some aspects, the processing system may determine whether at least one compensation example is still available, whether at least one training period or iteration is still needed, whether a defined amount of time or computational resources have been spent on training, and whether the compensated model exhibits the expected minimum level of accuracy, etc.

[0088] If the criteria are not met, method 400 returns to box 405. If the criteria are met, method 400 continues to box 455, where the processing system deploys the compensated model. As discussed above, deploying the compensated model may typically include any operations used to prepare or provide the model for inference (such as sending the parameters of the compensated model to the inference system, using one or more hardware components, etc.).

[0089] Example methods for adapting quantization compensation models

[0090] Figure 5 This is a flowchart depicting an example method 500 for adapting a quantization compensation model according to some aspects of this disclosure. In some aspects, method 500 comprises one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3 and / or Figure 4 The machine learning system under discussion is executed. In some respects, method 500 is aimed at... Figure 3 Box 320 provides additional details.

[0091] At box 505, the processing system selects the adaptation sample (e.g., from...). Figure 1 (The adapted data consists of 145 examples). Typically, the processing system can use any suitable criteria (including random or pseudo-random) to select adapted examples. In some respects, as discussed above, adapted examples may typically correspond to the target domain (e.g., data used to adapt the model to a specific user or entity), while the data used to train the baseline model may correspond to the source domain. In some respects, adapted examples may have one or more ground truth labels.

[0092] At box 510, the processing system uses a block of the quantized model (e.g., Figures 2A to 2C The system uses block 215 of the quantized model 125 to generate a first feature map. In some aspects, the specific data processed to generate the feature map may vary depending on the specific implementation. For example, if the block is the first block in the quantized model, the processing system can use that block to process selected adaptation samples to generate the first feature map. If the block is an internal block of the quantized model, the processing system can process the feature map generated by the previous components.

[0093] For example, if the compensated model uses a reference Figure 2A In the architecture discussed, the processing system can use the current block to process the feature maps generated by previous blocks of the quantized model in order to generate a first feature map. For example, if the compensated model uses a reference... Figure 2BIn the architecture discussed, the processing system can use the current block to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate the first feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use the current block of the quantized model to process the feature map generated by the previous block of the compensated model in order to generate the first feature map.

[0094] At box 515, the processing system determines whether there is a corresponding compensation block for the quantized block used to generate the first feature map at box 510. If not, method 500 proceeds to box 535. If there is a corresponding compensation block in the compensation model, method 500 continues to box 520.

[0095] At box 520, the processing system uses the corresponding block of the compensation model (e.g., Figures 2A to 2C The compensation model 210 (block 220) is used to generate the second feature map. In some respects, the specific data processed to generate the second feature map may vary depending on the specific implementation.

[0096] For example, if the compensated model uses a reference Figure 2A The architecture discussed allows the processing system to handle intermediate feature maps generated from corresponding blocks of the quantized model, as discussed above. For example, if the compensated model uses a reference... Figure 2B In the architecture discussed, the processing system can use compensation blocks to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate a second feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use compensation blocks to process the feature maps generated by the corresponding blocks of the quantized model (e.g., at box 510) in order to generate a second feature map.

[0097] At box 535, the processing system determines whether there is at least one more block in the quantized model. If so, method 500 returns to box 510 to generate a new feature map using the next block in the quantized model. If no more blocks remain, method 500 continues to box 540.

[0098] At box 540, the processing system optionally generates an end-to-end adaptation loss based on the output of the compensated model (e.g., the final output of the quantized model and / or the compensated model). In some aspects, as discussed above, the generation of the adaptation loss is further based on the ground truth labels of selected adaptation samples. For example, the processing system may compute a cross-entropy loss based on the output and ground truth (or use any other suitable loss algorithm). Method 500 then continues to box 545.

[0099] At box 545, the processing system updates one or more parameters of the compensation model (e.g., using backpropagation), as discussed above. In some respects, the parameters of the quantized machine learning model are static and remain unchanged during the update of the compensation model. While the illustrated example depicts stochastic gradient descent (e.g., refining the compensation model individually based on each fit example) for conceptual clarity, in some respects, the processing system may additionally or alternatively use batch gradient descent.

[0100] At box 550, the processing system determines whether one or more termination criteria are met. Typically, the termination criteria used may vary depending on the specific implementation. For example, in some aspects, the processing system may determine whether at least one adapted sample is still to be used, whether at least one training period or iteration is still pending, whether a defined amount of time or computational resources have been spent on training, whether the compensated model exhibits the expected minimum level of accuracy, etc.

[0101] If the criteria are not met, method 500 returns to box 505. If the criteria are met, method 500 continues to box 555, where the system deploys the adapted model. As discussed above, deploying the adapted model typically includes any operations used to prepare or provide the model for inference (such as sending the parameters of the adapted model to the inference system, using one or more hardware components, etc.).

[0102] While the illustrated method 500 describes the use of end-to-end loss to adapt the compensation model, in some respects the processing system may additionally or alternatively use block-by-block loss to train the adaptation model, as discussed above.

[0103] Example methods for generating output using a machine learning model with quantized compensation.

[0104] Figure 6 This is a flowchart depicting an example method 600 for generating output using a machine learning model with quantized compensation, according to some aspects of this disclosure. In some aspects, method 600 comprises one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3 , Figure 4 and / or Figure 5 The machine learning system under discussion is executed.

[0105] At box 605, system access input data is processed (e.g., Figures 2A to 2C The input data (205) lacks ground truth values ​​and is accessed for processing to generate predictions (e.g., continuous or categorical values). In some respects, the input data corresponds to the target domain (e.g., if the compensated model has been adapted to the target domain).

[0106] At box 610, the processing system uses a block of the quantized model (e.g., Figures 2A to 2C The system uses blocks 215 of the quantized model 125 to generate a first feature map. In some aspects, the specific data processed to generate the feature map may vary depending on the specific implementation. For example, if the block is the first block in the quantized model, the processing system can use that block to process the input data to generate the first feature map. If the block is an internal block of the quantized model, the processing system can process the feature map generated by the previous components.

[0107] For example, if the compensated model uses a reference Figure 2A In the architecture discussed, the processing system can use the current block to process the feature maps generated by previous blocks of the quantized model in order to generate a first feature map. For example, if the compensated model uses a reference... Figure 2B In the architecture discussed, the processing system can use the current block to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate the first feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use the current block of the quantized model to process the feature map generated by the previous block of the compensated model in order to generate the first feature map.

[0108] At box 615, the processing system determines whether there is a corresponding compensation block for the quantized block used to generate the first feature map at box 610. If not, method 600 proceeds to box 635. If there is a corresponding compensation block in the compensation model, method 600 continues to box 620.

[0109] At box 620, the processing system uses the corresponding block of the compensation model (e.g., Figures 2A to 2C The compensation model 210 (block 220) is used to generate the second feature map. In some respects, the specific data processed to generate the second feature map may vary depending on the specific implementation.

[0110] For example, if the compensated model uses a reference Figure 2A The architecture discussed allows the processing system to handle intermediate feature maps generated from corresponding blocks of the quantized model, as discussed above. For example, if the compensated model uses a reference... Figure 2B In the architecture discussed, the processing system can use compensation blocks to process the feature map generated by the previous aggregation operation (e.g., aggregation operation 230) to generate a second feature map. Furthermore, if the compensated model uses a reference... Figure 2C In the architecture discussed, the processing system can use compensation blocks to process the feature maps generated by the corresponding blocks of the quantized model (e.g., at box 610) in order to generate a second feature map.

[0111] At box 635, the processing system determines whether there is at least one more block in the quantized model. If so, method 600 returns to box 610 to generate a new feature map using the next block in the quantized model. If no more blocks remain, method 600 continues to box 640.

[0112] At box 640, the processing system returns the model output as a prediction of the input data. Typically, returning the model output may include providing the model output to an entity that provides the input data and / or requests the generation of a prediction (e.g., an application), outputting the prediction for display, or outputting the prediction to downstream components or systems.

[0113] Example methods for using machine learning models to compensate for quantization

[0114] Figure 7 This is a flowchart depicting an example method 700 for using a machine learning model to compensate for or at least adjust quantization, according to some aspects of this disclosure. In some aspects, method 700 consists of one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3 , Figure 4 , Figure 5 and / or Figure 6 The machine learning system under discussion is executed.

[0115] At box 705, a first machine learning model is accessed, which includes a first plurality of blocks. The first plurality of blocks are associated with a first precision and include the first block.

[0116] At box 710, the second machine learning model is accessed, which includes a second plurality of blocks associated with a second precision different from the first precision. In some respects, the second plurality of blocks includes a first block, and the first block in the second plurality of blocks corresponds to the first block in the first plurality of blocks.

[0117] In some respects, the second precision is higher than the first precision. In some respects, the second precision corresponds to a 16-bit width, and the first precision corresponds to a 4-bit width.

[0118] In some respects, the first machine learning model has a first size, the second machine learning model has a second size, and the second size is smaller than the first size.

[0119] In some respects, the first machine learning model is generated by quantizing a baseline machine learning model that has a higher accuracy than the first accuracy. In this case, the second machine learning model can be trained to adjust for the quantization error caused by the quantization of the baseline machine learning model.

[0120] At box 715, the first plurality of blocks of the first machine learning model and the second plurality of blocks of the second machine learning model are used to process the input to the first machine learning model. This processing may involve modifying the output of the first block in the first plurality of blocks based on the corresponding first block in the second plurality of blocks.

[0121] In some aspects, the first plurality of blocks comprises an ordered network of blocks, and the first plurality of blocks may further comprise a second block configured to receive a modified output of the first block as input and process the received input. In such aspects, processing the input using the first machine learning model may also include modifying the output of the second block of the first plurality of blocks using a corresponding second block of the second plurality of blocks.

[0122] At box 720, the output of the first machine learning model is provided based on this processing.

[0123] In some respects, the first machine learning model is accessed by a first circuit of the IC device, and the second machine learning model is accessed by a second circuit of the IC device that is different from the first circuit.

[0124] In some respects, the first machine learning model is trained on training data from the source domain, the second machine learning model is trained using adjusted data from the source domain, and the second machine learning model is trained without using the labels of the adjusted data.

[0125] In some respects, training the second machine learning model includes generating an adjusted loss for a first block of the second machine learning model based on: (i) a first feature map generated from the first block of the baseline machine learning model based on a first instance in the adjusted data; (ii) a second feature map generated from a quantized version of the first block of the baseline machine learning model based on the first instance, the quantized version corresponding to the first block of the first plurality of blocks; and (iii) a third feature map generated from the first block of the second plurality of blocks based on the first instance.

[0126] In some respects, training the second machine learning model includes generating an adjusted loss for the first block of the second plurality of blocks based on: (i) a first model output generated by the baseline machine learning model based on a first instance in the adjusted data; (ii) a second model output generated by the first machine learning model based on the first instance; and (iii) a third model output generated by the second machine learning model based on the first instance.

[0127] In some aspects, method 700 also includes adapting the second machine learning model to the target domain based on labeled adaptation data for the target domain. The first machine learning model may be frozen during the adaptation to the target domain.

[0128] Example methods for training machine learning models to compensate for quantization.

[0129] Figure 8 This is a flowchart depicting an example method 800 for training a machine learning model to compensate for or at least adjust quantization, according to some aspects of this disclosure. In some aspects, method 800 consists of one or more processing systems (such as those referenced above). Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3 , Figure 4 , Figure 5 , Figure 6 and / or Figure 7 The machine learning system under discussion is executed.

[0130] At box 805, access the first machine learning model, which includes a first plurality of blocks.

[0131] In some respects, each of the first plurality of blocks includes at least one of a layer of the first machine learning model or a transformer of the first machine learning model.

[0132] At box 810, a second machine learning model is generated by quantizing the first machine learning model, the second machine learning model comprising a second plurality of blocks.

[0133] At box 815, a third machine learning model is trained to adjust for the quantization of the first machine learning model. The third machine learning model includes a third set of blocks.

[0134] In some respects, the first machine learning model is trained on training data from the source domain, the third machine learning model is trained using adjusted data from the source domain, and the third machine learning model is trained without using the labels of the adjusted data.

[0135] In some respects, training the third machine learning model includes generating an adjusted loss for a first block of the third plurality of blocks based on: (i) a first feature map generated from the first block of the first plurality of blocks based on a first instance in the adjusted data; (ii) a second feature map generated from the first block of the second plurality of blocks based on the first instance, wherein the first block from the second plurality of blocks includes a quantized version of the first block from the first plurality of blocks; and (iii) a third feature map generated from the first block of the third plurality of blocks based on the first instance, wherein the first block of the third plurality of blocks corresponds to the first block of the second plurality of blocks.

[0136] In some respects, training the third machine learning model includes generating an adjusted loss for a first block of the third plurality of blocks based on: (i) a first model output generated by the first machine learning model based on a first instance in the adjusted data; (ii) a second model output generated by the second machine learning model based on the first instance; and (iii) a third model output generated by the third machine learning model based on the first instance.

[0137] In some respects, the parameters of the second machine learning model are encoded using a first value representation, the parameters of the third machine learning model are encoded using a second value representation, and the second value representation has higher precision than the first value representation.

[0138] At box 820, the second and third machine learning models are deployed for inference.

[0139] In some aspects, method 800 also includes adapting the third machine learning model to the target domain based on labeled adaptation data for the target domain, wherein the second machine learning model is frozen during adaptation to the target domain.

[0140] In some respects, deploying the second machine learning model and the third machine learning model for inference includes deploying the second machine learning model to execute on the first hardware component and deploying the third machine learning model to execute on the second hardware component.

[0141] Example processing system for machine learning

[0142] Figure 9 An example processing system 900 is depicted, which is configured to perform various aspects of this disclosure, including, for example, regarding... Figures 1 to 8The techniques and methods described herein. In some aspects, the processing system 900 may correspond to a training system. For example, the processing system 900 may correspond to a device training a machine learning model, a quantized machine learning model, training a compensated machine learning model, adapting a compensated machine learning model, and / or using the compensated and / or adapted machine learning model for inference. Although depicted as a single system for clarity of concept, in some aspects, as discussed above, the operations described below with respect to the processing system 900 may be distributed across any number of devices or systems.

[0143] The processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed at the CPU 902 may be loaded, for example, from program memory associated with the CPU 902 or from a memory partition (e.g., a partition of memory 924).

[0144] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, a multimedia component 910 (e.g., a multimedia processing unit), and a wireless connectivity component 912.

[0145] NPUs (such as the NPU 908) are typically configured to implement dedicated circuitry for the control and arithmetic logic used to execute machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0146] NPUs such as the NPU 908 are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0147] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0148] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0149] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0150] In some implementations, the NPU 908 is part of one or more of the CPU 902, GPU 904, and / or DSP 906.

[0151] In some examples, the wireless connectivity component 912 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Range Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 912 is also connected to one or more antennas 914.

[0152] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0153] The processing system 900 may also include one or more input and / or output devices 922, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0154] In some examples, one or more processors in the processing system 900 may be based on the ARM or RISC-V instruction set.

[0155] The processing system 900 also includes memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 924 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 900.

[0156] Specifically, in this example, memory 924 includes a training component 924A, a quantization component 924B, a compensation component 924C, and an adaptation component 924D. Although not depicted in the illustrated example, memory 924 may also include other components, such as an inference component, to generate output predictions based on the processed model input using the compensated machine learning model, as discussed above. Although for conceptual clarity... Figure 9 The components are depicted as discrete parts, but in various respects, the illustrated components (and other undescribed components) can be implemented together or separately.

[0157] As shown in the figure, memory 924 also includes a basic model parameter set 924E (e.g., parameters of a baseline machine learning model, such as...). Figure 1 The machine learning model 115) and the compensated model parameter set 924F (e.g., Figure 1 The parameters of the compensated model 140 and / or the adapted model 155). Although not depicted in the illustrated example, the memory 924 may also include other data, such as training data (e.g., Figure 1 Training data 105), compensation data (e.g., Figure 1 Compensation data 130) and / or adaptation data (e.g., Figure 1 Adaptation data 145).

[0158] The processing system 900 also includes a training circuit 926, a quantization circuit 927, a compensation circuit 928, and an adaptation circuit 929. The depicted circuits and others not depicted (such as inference circuits) can be configured to perform various aspects of the techniques described herein.

[0159] Training component 924A and / or training circuit 926 (which may correspond to Figure 1 The training component 110 can be used to train a baseline machine learning model (e.g., to learn basic model parameters 924E), as discussed above. For example, the training component 924A and / or the training circuit 926 can use training data to learn parameters encoded in a relatively high-precision format, such as a hexadecimal representation.

[0160] Quantization component 924B and / or quantization circuit 927 (which may correspond to) Figure 1 The quantization component 120 can be used to quantize the trained baseline machine learning model (e.g., to generate...). Figure 1 and Figures 2A to 2C The quantized model 125 is as discussed above. For example, quantization component 924B and / or quantization circuit 927 can quantize the model to generate a set of parameters encoded in a relatively low-precision format, such as four-bit representation.

[0161] Compensation component 924C and / or compensation circuit 928 (which may correspond to Figure 1 The compensation component 135 can be used to train the compensation model (e.g., Figures 2A to 2C The compensation model 210 is as discussed above. For example, the compensation component 924C and / or the compensation circuit 928 can use the compensation data to learn the parameters for the compensation model. In some respects, as discussed above, the parameters of the compensation model are encoded in a relatively high-precision format, such as a 16-bit representation.

[0162] Adapter component 924D and / or adapter circuit 929 (which may correspond to) Figure 1 The adapter component 150 can be used to adapt the compensated model (e.g., to generate...). Figure 1 The adapted model 155 is as discussed above. For example, the adapter component 924D and / or the adapter circuit 929 can use the adapter data to update or refine the parameters of the compensation model.

[0163] Although Figure 9 For clarity, the training circuit 926, quantization circuit 927, compensation circuit 928, and adaptation circuit 929 are depicted as separate components and circuits. They may be implemented together or separately in other processing devices of the processing system 900, such as within the CPU 902, GPU 904, DSP 906, and NPU 908.

[0164] Typically, the processing system 900 and / or its components can be configured to perform the methods described herein.

[0165] It is worth noting that, in other aspects, such as when the processing system 900 is a server computer, various aspects of the processing system 900 may be omitted. For example, in other aspects, the multimedia component 910, the wireless connectivity component 912, the sensor processing unit 916, the ISP 918, and / or the navigation processor 920 may be omitted. Furthermore, the various aspects of the processing system 900 may be distributed among multiple devices.

[0166] Example Terms

[0167] Specific implementation examples are described in the following numbered clauses: Clause 1: A method comprising: accessing a first machine learning model, the first machine learning model including a first plurality of blocks, the first plurality of blocks being associated with and including a first precision; accessing a second machine learning model, the second machine learning model including a second plurality of blocks, the second plurality of blocks being associated with a second precision different from the first precision, wherein: the second plurality of blocks include a first block; and the first block in the second plurality of blocks corresponds to the first block in the first plurality of blocks; processing an input to the first machine learning model using the first plurality of blocks of the first machine learning model and the second plurality of blocks of the second machine learning model, wherein the processing includes modifying the output of the first block in the first plurality of blocks based on the corresponding first block in the second plurality of blocks; and providing an output of the first machine learning model based on the processing.

[0168] Clause 2: The method according to Clause 1, wherein the second precision is higher than the first precision.

[0169] Clause 3: The method according to Clause 2, wherein the second precision corresponds to a 16-bit width and the first precision corresponds to a 4-bit width.

[0170] Clause 4: The method according to any one of Clauses 1 to 3, wherein: the first machine learning model has a first size, the second machine learning model has a second size, and the second size is smaller than the first size.

[0171] Clause 5: The method according to any one of Clauses 1 to 4, wherein: the first machine learning model is accessed by a first circuit of an integrated circuit (IC) device, and the second machine learning model is accessed by a second circuit of the IC device, which is different from the first circuit.

[0172] Clause 6: The method according to any one of Clauses 1 to 5, wherein: the first machine learning model is generated by quantizing a baseline machine learning model having a baseline accuracy higher than the first accuracy, and the second machine learning model is trained to adjust for quantization errors caused by the quantization of the baseline machine learning model.

[0173] Clause 7: The method according to any one of Clauses 1 to 6, wherein: the first plurality of blocks comprises an ordered network of blocks, the first plurality of blocks further comprises a second block, the second block being configured to receive a modified output of the first block of the first plurality of blocks as input and to process the received input, and processing the input using a quantized machine learning model further comprises modifying the output of the second block of the first plurality of blocks using a corresponding second block of the second plurality of blocks.

[0174] Clause 8: The method according to any one of Clauses 1 to 7, wherein: the first machine learning model is trained based on training data from the source domain, the second machine learning model is trained using adjusted data from the source domain, and the second machine learning model is trained without using labels on the adjusted data.

[0175] Clause 9: The method according to Clause 8, wherein training the second machine learning model includes generating an adjusted loss for a first block of the second machine learning model based on: (i) a first feature map generated by the first block of the baseline machine learning model based on a first instance in the adjusted data; (ii) a second feature map generated by a quantized version of the first block of the baseline machine learning model based on the first instance, the quantized version corresponding to the first block of the first plurality of blocks; and (iii) a third feature map generated by the first block of the second plurality of blocks based on the first instance.

[0176] Clause 10: The method according to any one of Clauses 8 to 9, wherein training the second machine learning model includes generating an adjusted loss for a first block of the second plurality of blocks based on: (i) a first model output generated by a baseline machine learning model based on a first example in the adjusted data; (ii) a second model output generated by the first machine learning model based on the first example; and (iii) a third model output generated by the second machine learning model based on the first example.

[0177] Clause 11: The method according to any one of Clauses 8 to 10, the method further comprising: adapting the second machine learning model to the target domain based on labeled adaptation data for the target domain, wherein the first machine learning model is frozen during adaptation to the target domain.

[0178] Clause 12: A method comprising: accessing a first machine learning model, the first machine learning model including a first plurality of blocks; generating a second machine learning model by quantizing the first machine learning model, the second machine learning model including a second plurality of blocks; training a third machine learning model for adjusting the quantization of the first machine learning model, the third machine learning model including a third plurality of blocks; and deploying the second machine learning model and the third machine learning model for inference.

[0179] Clause 13: The method according to Clause 12, wherein each of the first plurality of blocks comprises at least one of a layer of the first machine learning model or a transformer of the first machine learning model.

[0180] Clause 14: The method according to any one of Clauses 12 to 13, wherein: the first machine learning model is trained based on training data from the source domain, the third machine learning model is trained using adjusted data from the source domain, and the third machine learning model is trained without using labels on the adjusted data.

[0181] Clause 15: The method according to Clause 14, wherein training the third machine learning model comprises generating an adjusted loss for a first block of the third plurality of blocks based on: (i) a first feature map generated by the first block of the first plurality of blocks based on a first instance in the adjusted data; (ii) a second feature map generated by the first block of the second plurality of blocks based on the first instance, wherein the first block from the second plurality of blocks includes a quantized version of the first block from the first plurality of blocks; and (iii) a third feature map generated by the first block of the third plurality of blocks based on the first instance, wherein the first block of the third plurality of blocks corresponds to the first block of the second plurality of blocks.

[0182] Clause 16: The method according to any one of Clauses 14 to 15, wherein training the third machine learning model comprises generating an adjusted loss for a first block of the third plurality of blocks based on: (i) a first model output generated by the first machine learning model based on a first example in the adjusted data; (ii) a second model output generated by the second machine learning model based on the first example; and (iii) a third model output generated by the third machine learning model based on the first example.

[0183] Clause 17: The method according to any one of Clauses 12 to 16, the method further comprising: adapting the third machine learning model to the target domain based on labeled adaptation data for the target domain, wherein the second machine learning model is frozen during adaptation to the target domain.

[0184] Clause 18: The method according to any one of Clauses 12 to 17, wherein: the parameters of the second machine learning model are encoded using a first value representation, the parameters of the third machine learning model are encoded using a second value representation, and the second value representation has a higher precision than the first value representation.

[0185] Clause 19: The method according to any one of Clauses 12 to 18, wherein deploying the second machine learning model and the third machine learning model for inference includes deploying the second machine learning model to execute on a first hardware component and deploying the third machine learning model to execute on a second hardware component.

[0186] Clause 20: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 19.

[0187] Clause 21: A processing system comprising components for performing the method according to any one of Clauses 1 to 19.

[0188] Clause 22: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 19.

[0189] Clause 23: A computer program product embodied on a computer-readable storage medium, the computer-readable storage medium including code for performing the method according to any one of Clauses 1 to 19.

[0190] Additional Notes

[0191] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0192] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0193] As used in this article, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0194] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0195] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0196] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC §112(f) unless it is expressly referred to herein by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known thereafter, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: One or more memories, the one or more memories including computer-executable instructions; and One or more processors, the one or more processors being configured to execute the computer-executable instructions and cause the processing system to: Access a first machine learning model, the first machine learning model comprising a first plurality of blocks, the first plurality of blocks being associated with a first precision and including the first block; Access a second machine learning model, which includes a second plurality of blocks, and these blocks are associated with a second precision different from the first precision, wherein: The second plurality of blocks includes the first block; and The first block in the second plurality of blocks corresponds to the first block in the first plurality of blocks; The input to the quantized machine learning model is processed using the first plurality of blocks of the first machine learning model and the second plurality of blocks of the second machine learning model, wherein, in order to process the input, the one or more processors are configured to execute the computer-executable instructions and cause the processing system to modify the output of the first block of the first plurality of blocks based on a corresponding first block of the second plurality of blocks; and The output of the first machine learning model is provided based on the processing described above.

2. The processing system according to claim 1, wherein the second precision is higher than the first precision.

3. The processing system according to claim 2, wherein the second precision corresponds to a 16-bit width, and wherein the first precision corresponds to a 4-bit width.

4. The processing system according to claim 1, wherein: The first machine learning model has a first size; The second machine learning model has a second size; and The second size is smaller than the first size.

5. The processing system according to claim 1, wherein: The first machine learning model is accessed by a first circuit of an integrated circuit (IC) device; and The second machine learning model is accessed by a second circuit of the IC device, which is different from the first circuit.

6. The processing system according to claim 1, wherein: The first machine learning model is generated by quantizing a baseline machine learning model that has a higher accuracy than the first precision. The second machine learning model is trained to adjust for the quantization error caused by the quantization of the baseline machine learning model.

7. The processing system according to claim 1, wherein: The first plurality of blocks comprise an ordered network of blocks. The first plurality of blocks further includes a second block, the second block being configured to receive the modified output of the first block as input and to process the received input, and In order to process the input, the one or more processors are configured to further execute the computer-executable instructions and cause the processing system to modify the output of the second block in the first plurality of blocks using a corresponding second block in the second plurality of blocks.

8. The processing system according to claim 1, wherein: The first machine learning model is trained based on training data from the source domain. The second machine learning model is trained using adjusted data from the source domain, and The second machine learning model is trained without using the labels on the adjusted data.

9. The processing system according to claim 8, wherein, To train the second machine learning model, the one or more processors are configured to execute the computer-executable instructions and cause the processing system to generate an adjusted loss for a first block of the second machine learning model based on the following: (i) A first feature map generated by the first block of the baseline machine learning model based on the first instance in the adjusted data; (ii) A second feature map generated based on the first example from a quantized version of the first block of the baseline machine learning model, the quantized version corresponding to the first block among the first plurality of blocks; and (iii) A third feature map generated by the first block of the second plurality of blocks based on the first example.

10. The processing system of claim 8, wherein, for training the second machine learning model, the one or more processors are configured to execute the computer-executable instructions and cause the processing system to generate an adjustment loss for the first block of the second plurality of blocks based on: (i) The first model output generated by the baseline machine learning model based on the first instance in the adjusted data; (ii) The output of the second model generated by the first machine learning model based on the first example; and (iii) The output of the third model generated by the second machine learning model based on the first example.

11. The processing system according to claim 8, wherein: The one or more processors are configured to further execute the computer-executable instructions and cause the processing system to adapt the second machine learning model to the target domain based on labeled adaptation data for the target domain; and The first machine learning model is frozen during adaptation to the target domain.

12. A processor-implemented method, the processor-implemented method comprising: Access a first machine learning model, the first machine learning model comprising a first plurality of blocks, the first plurality of blocks being associated with a first precision and including the first block; Access a second machine learning model, which includes a second plurality of blocks, and these blocks are associated with a second precision different from the first precision, wherein: The second plurality of blocks includes the first block; and The first block in the second plurality of blocks corresponds to the first block in the first plurality of blocks; The input to the first machine learning model is processed using the first plurality of blocks of the first machine learning model and the second plurality of blocks of the second machine learning model, wherein the processing includes modifying the output of the first block of the first plurality of blocks based on a corresponding first block of the second plurality of blocks; and The output of the first machine learning model is provided based on the processing described above.

13. The processor-implemented method of claim 12, wherein the second precision is higher than the first precision.

14. The processor-implemented method of claim 13, wherein the second precision corresponds to a 16-bit width, and wherein the first precision corresponds to a 4-bit width.

15. The processor-implemented method according to claim 12, wherein: The first machine learning model has a first size; The second machine learning model has a second size; and The second size is smaller than the first size.

16. The processor-implemented method according to claim 12, wherein: The first machine learning model is accessed by a first circuit of an integrated circuit (IC) device; and The second machine learning model is accessed by a second circuit of the IC device, which is different from the first circuit.

17. The processor-implemented method according to claim 12, wherein: The first machine learning model is generated by quantizing a baseline machine learning model that has a higher accuracy than the first precision. The second machine learning model is trained to adjust for the quantization error caused by the quantization of the baseline machine learning model.

18. The processor-implemented method according to claim 12, wherein: The first plurality of blocks comprise an ordered network of blocks. The first plurality of blocks further includes a second block, the second block being configured to receive the modified output of the first block as input and to process the received input, and Using the first machine learning model to process the input also includes modifying the output of the second block in the first plurality of blocks using the corresponding second block in the second plurality of blocks.

19. The processor-implemented method according to claim 12, wherein: The first machine learning model is trained based on training data from the source domain. The second machine learning model is trained using adjusted data from the source domain, and The second machine learning model is trained without using the labels on the adjusted data.

20. The processor-implemented method of claim 19, wherein training the second machine learning model comprises generating an adjusted loss for a first block of the second machine learning model based on the following: (i) A first feature map generated by the first block of the baseline machine learning model based on the first instance in the adjusted data; (ii) A second feature map generated based on the first example from a quantized version of the first block of the baseline machine learning model, the quantized version corresponding to the first block among the first plurality of blocks; and (iii) A third feature map generated by the first block of the second plurality of blocks based on the first example.

21. The processor-implemented method of claim 19, wherein training the second machine learning model comprises generating an adjusted loss for the first block of the second plurality of blocks based on various terms: (i) The first model output generated by the baseline machine learning model based on the first instance in the adjusted data; (ii) The output of the second model generated by the first machine learning model based on the first example; and (iii) The output of the third model generated by the second machine learning model based on the first example.

22. The processor-implemented method according to claim 19, further comprising: The second machine learning model is adapted to the target domain based on labeled adaptation data for the target domain, wherein the first machine learning model is frozen during the adaptation to the target domain.

23. A processor-implemented method, the processor-implemented method comprising: Access the first machine learning model, which includes a first plurality of blocks; A second machine learning model is generated by quantizing the first machine learning model, the second machine learning model comprising a second plurality of blocks; A third machine learning model is trained to adjust for the quantization of the first machine learning model, the third machine learning model comprising a third plurality of blocks; as well as Deploy the second machine learning model and the third machine learning model for inference.

24. The processor-implemented method of claim 23, wherein each of the first plurality of blocks comprises at least one of a layer of the first machine learning model or a transformer of the first machine learning model.

25. The processor-implemented method according to claim 23, wherein: The first machine learning model is trained based on training data from the source domain. The third machine learning model is trained using adjusted data from the source domain, and The third machine learning model is trained without using the labels on the adjusted data.

26. The processor-implemented method of claim 25, wherein training the third machine learning model comprises generating an adjusted loss for the first block of the third plurality of blocks based on the following: (i) A first feature map generated by a first block of the first plurality of blocks based on a first instance in the adjusted data; (ii) A second feature map generated by the first block of the second plurality of blocks based on the first sample, wherein the first block from the second plurality of blocks includes a quantized version of the first block from the first plurality of blocks; and (iii) A third feature map generated by the first block of the third plurality of blocks based on the first example, wherein the first block of the third plurality of blocks corresponds to the first block of the second plurality of blocks.

27. The processor-implemented method of claim 25, wherein training the third machine learning model comprises generating an adjusted loss for the first block of the third plurality of blocks based on the following: (i) The first model output generated by the first machine learning model based on the first instance in the adjusted data; (ii) The output of the second model generated by the second machine learning model based on the first example; and (iii) The output of the third model generated by the third machine learning model based on the first example.

28. The processor-implemented method according to claim 25, further comprising: The third machine learning model is adapted to the target domain based on labeled adaptation data for the target domain, wherein the second machine learning model is frozen during the adaptation to the target domain.

29. The processor-implemented method according to claim 23, wherein: The parameters of the second machine learning model are encoded using the first value. The parameters of the third machine learning model are encoded using a second value representation, and The second value represents a higher precision than the first value.

30. The processor-implemented method of claim 23, wherein deploying the second machine learning model and the third machine learning model for inference includes deploying the second machine learning model to execute on a first hardware component and deploying the third machine learning model to execute on a second hardware component.

Images