Outlier removal for transformer network quantization

Abstract

The invention relates to outlier removal for transformer network quantization. An example device will clip (494) a value of an activation associated with a layer of a floating point version of a machine learning model (410) to determine a clipped value of the activation, the value of the activation based on calibration data (455) applied to the floating point version of the machine learning model (410). The example device will also determine (496) a quantization factor (460) using the clipped value of the activation to quantify an activation associated with a corresponding layer of a fixed point version of the machine learning model (415). The example apparatus will further configure (425) the fixed point version of the machine learning model (415) on a device (430) using the quantization factor (460).

Description

[0001] Cross-references to related applications

[0002] This patent application claims the benefit and priority of Indian Provisional Patent Application No. 202441064556, filed on August 26, 2024, which is hereby incorporated herein by reference in its entirety.

[0003] This patent application is also incorporated, by reference in its entirety, of the following commonly assigned patent applications: (i) U.S. Patent Publication No. 2024 / 0036816, published February 1, 2024, entitled “Systems and Methods for Identifying Scaling Factors for Deep Neural Networks”; (ii) U.S. Patent Publication No. 2024 / 0062059, published February 22, 2024, entitled “Neural Network Layer Optimization”; (iii) U.S. Patent Application No. 18 / 408,351, filed January 9, 2024, entitled “Quantization of Neural Networks”; and (iv) U.S. Patent Application No. 18 / 917,252, filed October 16, 2024, entitled “Optimization of Transformer Encoders”. Technical Field

[0004] This specification generally relates to machine learning, and more specifically, to outlier removal for transformer network quantization. Background Technology

[0005] For example, deep machine learning models using deep neural networks (DNNs) are used for various computer vision tasks, such as object detection, image segmentation, and image classification. Transformer networks are a type of DNN that utilizes a transformer encoder to perform various tasks, such as computer vision, language processing, and audio processing. The input to a transformer network includes sensor data, such as data from cameras and other image sensors, LiDAR sensors, radar sensors, etc., which can support applications such as machine vision, industrial inspection, advanced driver assistance, and autonomous driving. The output of a transformer network is task-dependent. For example, if a transformer network is configured to perform image classification, its input will contain image data, and its output will contain the classification of the input image.

[0006] Machine learning models, such as converter networks and DNNs, can be trained on floating-point implementations that utilize floating-point operations to process floating-point data. Such floating-point machine learning models can be designed to be implemented on cloud-based platforms or other high-performance target platforms with sufficient processing and memory capabilities to perform floating-point operations on the model. However, embedded devices, such as embedded system-on-a-chip (SoC) devices, are preferred target platforms for deploying trained machine learning models. Such embedded devices may have limited processor and memory capabilities and are therefore designed to perform fixed-point operations on fixed-point data. Model quantization refers to the process of converting the floating-point implementation of a machine learning model into a corresponding fixed-point implementation, which involves converting the precision of the model's weights and activations from floating-point precision to fixed-point precision. Summary of the Invention

[0007] For methods and apparatus for performing outlier removal for converter network quantization, the example non-transitory computer-readable media described herein contains example computer-readable instructions to cause at least one processor circuitry to clip values ​​of activations associated with layers of a floating-point version of a machine learning model to determine clipped values ​​of the activations, said values ​​being based on calibration data applied to the floating-point version of the machine learning model. The example instructions further cause one or more of the at least one processor circuitry to determine a quantization factor using the clipped values ​​of the activations to quantize activations associated with corresponding layers of a fixed-point version of the machine learning model. The example instructions further cause one or more of the at least one processor circuitry to configure the fixed-point version of the machine learning model on an apparatus using the quantization factor.

[0008] For methods and apparatuses used to perform outlier removal for converter network quantization, the example apparatus described herein includes an interface circuitry, machine-readable instructions, and at least one processor circuitry programmed based on the machine-readable instructions to clip values ​​of activations associated with layers of a floating-point version of a machine learning model to determine clipped values ​​of the activations, the activation values ​​being based on calibration data applied to the floating-point version of the machine learning model. One or more of the at least one processor circuitry further uses the clipped values ​​of the activations to determine a quantization factor to quantize activations associated with corresponding layers of a fixed-point version of the machine learning model. One or more of the at least one processor circuitry further uses the quantization factor to configure the fixed-point version of the machine learning model on the apparatus.

[0009] For methods and apparatuses used to perform outlier removal for converter network quantization, the example methods described herein include clipping values ​​of activations associated with layers of a floating-point version of a machine learning model to determine clipped values ​​of the activations, the values ​​of which are based on calibration data applied to the floating-point version of the machine learning model. The example methods also include using the clipped values ​​of the activations to determine a quantization factor to quantize activations associated with corresponding layers of a fixed-point version of the machine learning model. The example methods further include using the quantization factor to configure the fixed-point version of the machine learning model on an apparatus. Attached Figure Description

[0010] Figures 1A to 1B This describes the example operating environment that includes the example transformer network.

[0011] Figure 2 This section describes an example method for implementing a transformer network.

[0012] Figures 3A to 3C The description illustrates a system configured to perform image classification using an example transformer network.

[0013] Figure 4 This is a block diagram of an example environment where the example model quantizer circuitry operates to quantize a trained floating-point machine learning model.

[0014] Figure 5 Explain the example of quantization operation.

[0015] Figure 6 Explanation by Figure 4 The example quantization factor is determined by the model quantizer circuit system.

[0016] Figure 7 This describes the example activation data observed at each layer of the example machine learning model.

[0017] Figure 8 Explanation by Figure 4 The example function used by the model quantizer circuit system to perform outlier removal to quantize a trained floating-point machine learning model.

[0018] Figure 9 Explanation by Figure 4 The model quantizer circuit system performs an example type of quantization.

[0019] Figures 10 to 11 It is a flowchart representing an example machine-readable instruction or example operation, which can be used... Figure 4The example programmable circuit system implementation of the model quantizer circuit system performs at least one of the following operations: executes example machine-readable instructions or example operations, instantiates example machine-readable instructions or example operations, or implements example machine-readable instructions or example operations.

[0020] Figure 12 Explanation by Figure 4 Example model quantization performance results of the model quantizer circuit system implementation.

[0021] Figure 13 illustrate Figure 4 Examples of advantages of model quantizer circuit systems.

[0022] Figure 14 This is a block diagram of an example processing platform containing a programmable circuit system configured to execute, instantiate, or perform example machine-readable instructions or execute... Figures 10 to 11 Example operations to implement Figure 4 The model quantizer circuit system.

[0023] Figure 15 yes Figure 14 A block diagram of an example implementation of a programmable circuit system.

[0024] Figure 16 yes Figure 14 A block diagram of another example implementation of a programmable circuit system.

[0025] Figure 17 This is a block diagram of an example software / firmware / instruction distribution platform (e.g., one or more servers) used to distribute software, instructions, or firmware (e.g., corresponding to...) Figures 10 to 11 Example machine-readable instructions) are distributed to client devices associated with end users or consumers (e.g., for licensing, selling, or using), retailers (e.g., for selling, reselling, licensing, or sublicensing), or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers or other end users (e.g., direct purchase customers).

[0026] The diagrams are not necessarily to scale. Generally, the same reference numerals in the diagrams and this description refer to features and / or parts that are (functionally and / or structurally) the same or similar. Although the diagrams show areas with clearly defined lines and boundaries, some or all of these lines and boundaries may be idealized. In reality, boundaries or lines may be unobservable, mixed, or irregular. Detailed Implementation

[0027] This paper discloses a technique for improving the accuracy of quantized machine learning models (e.g., quantized transformer networks) by removing outliers during model quantization. Transformer networks are a type of deep learning network designed for a variety of applications. For example, trainable transformer networks can perform image segmentation, image classification, object detection, language processing, or other deep learning tasks. Determining a trained machine learning model (e.g., a trained transformer network) may involve training a floating-point implementation of the machine learning model, which includes weights and activations with floating-point precision. Quantizing the trained machine learning model involves converting the model's weights and activations from floating-point precision to fixed-point precision to generate a corresponding fixed-point implementation of the machine learning model.

[0028] In at least some examples, the fixed-point weights and activations of the fixed-point machine learning model are represented using fewer bits (e.g., 8 bits in some examples) than the floating-point weights and activations of the floating-point machine learning model (e.g., 32 bits in some examples). Therefore, the quantized fixed-point machine learning model can exhibit increased error and / or decreased accuracy compared to the original floating-point machine learning model. This is because the fixed-point weights and activations of the fixed-point machine learning model have fewer bits to represent the range of floating-point weights and activations of the floating-point machine learning model. Moreover, the presence of outliers in the observed weights and activations of the floating-point machine learning model can further increase the range of values ​​to be represented by the weights and activations of the fixed-point machine learning model, which may contribute to a further increase in model error and / or a decrease in model accuracy.

[0029] As described in detail below, the example model quantization technique disclosed herein removes outliers from the weights and activations of a floating-point machine learning model observed during quantization. By removing such outliers, the range of values ​​to be represented by the weights and activations of a fixed-point machine learning model is reduced, which can lead to improved model error and / or model accuracy relative to other model quantization techniques.

[0030] Turn to the attached diagram. Figure 1A This describes an example operating environment 100 that can be configured to perform a converter network. For example, operating environment 100 may represent a system configured to perform computer vision tasks (e.g., image classification, object detection, or another task). Operating environment 100 can be implemented in various use cases, such as automotive, industrial, robotics, building automation, language processing, power electronics, autonomous systems, radar, image processing, audio processing, or other applications requiring computer vision and / or processing of other data (e.g., text data, speech data, audio signals, radar signals, etc.). Operating environment 100 includes, but is not limited to, sensors 101 and processing circuitry 103.

[0031] Example sensor 101 represents a sensor configured to collect input data for performing a converter network. For example, sensor 101 may represent a camera, radar device, or another sensor configured to collect sensor data for performing converter network 105. In embodiments, sensor 101 is configured to collect image data of the environment or other sensor data. For example, sensor 101 may represent a camera mounted on a vehicle and configured to collect image data of the environment surrounding the vehicle. For illustrative purposes, image data will be discussed herein. This is not intended to limit the application of the proposed techniques, but rather to provide examples. Sensor 101 is coupled to processing circuitry system 103 and configured to output image data to processing circuitry system 103.

[0032] Example processing circuitry system 103 represents a circuitry system configured to execute a converter network. For example, processing circuitry system 103 may represent a central processing unit (CPU), application-specific integrated circuit (ASIC), digital signal processor (DSP), microcontroller unit (MCU), graphics processing unit (GPU), tensor processing unit (TPU), or another general-purpose processor (GPP), etc. Processing circuitry system 103 includes, but is not limited to, converter network 105.

[0033] Example transformer network 105 represents a deep learning network configured to perform a specified task. The input to transformer network 105 includes sensor data, while the output of transformer network 105 is task-dependent. For example, if transformer network 105 is configured to perform image classification, sensor 101 may collect image data of the environment and provide the image data to transformer network 105. In response, transformer network 105 may output a classification of the image data. Transformer network 105 includes encoder 106.

[0034] Example encoder 106 represents a converter encoder configured to employ an attention mechanism to perform a task configured to be performed by converter network 105. An attention mechanism describes a technique for determining the relative importance of features captured by image data from sensor 101. In an embodiment, encoder 106 utilizes a multi-head attention mechanism to perform the converter network 105. A multi-head attention mechanism represents a type of attention mechanism that allows the converter encoder to simultaneously analyze different features of the input data. Encoder 106 includes, but is not limited to, example block 108, example multi-head attention block (MHAB) 110, example block 112, example MHAB 114, example block 116, example block 118, and example control logic 120.

[0035] Block 108 represents a processing block configured to generate input data for performing the multi-head attention mechanism of encoder 106. For example, block 108 may be configured to generate input data for performing MHAB 110. In an embodiment, to generate input data for performing MHAB 110, block 108 is configured to embed image data from sensor 101 into multiple image matrices. For example, block 108 may receive image data from sensor 101, divide the image data into multiple image blocks, embed these image blocks into an equal number of image matrices, and supply said number of image matrices as input to MHAB 110. In response, MHAB 110 is configured to apply weight values ​​to the multiple image matrices to generate input data for performing the multi-head attention mechanism of MHAB 110. For example, MHAB 110 may apply key weights, query weights, and value weights to each image matrix to generate key data, query data, and value data for each image matrix.

[0036] The query data for an image matrix represents a matrix describing the viewpoint of the image matrix within the input image. For example, the query data could indicate that the image matrix represents the first image matrix of the input image. The key data for an image matrix represents a matrix describing the relationships between the image matrix and other image matrices within the input image. For example, the key data could indicate that the image matrix includes data related to the data of other image matrices in the input image. The value data for an image matrix represents a matrix describing the actual data of the image matrix. For example, the value data could store the data of the image matrix.

[0037] MHAB 110 represents a processing block configured to perform a series of attention-based operations on the query data, key data, and value data of each image matrix. For example, MHAB 110 can be configured to compute scaled dot-product attention for each image matrix of the input image. Scaled dot-product attention represents the attention mechanism used to determine the normalized attention score of the image matrix. In an implementation, to determine the scaled dot-product attention for each image matrix, MHAB 110 performs a series of layers, such that the first layer represents a matrix multiplication layer, the second layer represents a SoftMax layer, and the third layer represents another matrix multiplication layer, as will be discussed later. Figure 1B Let's discuss this in detail.

[0038] The output of MHAB 110 contains a final attention score matrix. This final attention score matrix represents a matrix storing the final attention scores for each image matrix of the original input image. For example, if the input image is divided into four image matrices, the output of MHAB 110 represents a matrix storing the final attention scores for all four image matrices. In this implementation, MHAB 110 is configured to provide its output to block 112.

[0039] Block 112 represents a processing block configured to generate input data for performing another multi-head attention mechanism of encoder 106. For example, block 112 may be configured to generate input data for performing MHAB 114. In an embodiment, to generate input data for performing MHAB 114, block 112 is configured to normalize the output of MHAB 110 and supply the normalized output to MHAB 114. For example, block 112 may include a final attention score matrix of MHAB 110 configured to normalize it and supply the normalized matrix to a normalization layer of MHAB 114. In response, MHAB 114 is configured to apply weight values ​​to the normalized matrix to generate input data for performing the multi-head attention mechanism of MHAB 114. For example, MHAB 114 may apply key weights, query weights, and value weights to the normalized matrix to generate key data, query data, and value data of the normalized matrix.

[0040] MHAB 114 represents a processing block configured to perform a series of attention-based operations on query data, key data, and value data of a normalized attention matrix. For example, MHAB 114 may also include multiple layers for computing scaled dot product attention, such that a first layer represents a matrix multiplication layer, a second layer represents a SoftMax layer, and a third layer represents another matrix multiplication layer. The output of MHAB 114 contains a final attention score matrix. The final attention score matrix of MHAB 114 represents a matrix of the final attention scores of the output of storage block 112. In an embodiment, MHAB 114 is configured to provide its output to block 116.

[0041] Block 116 represents another processing block configured to generate input data for performing another multi-head attention mechanism of encoder 106. For example, block 116 may represent block 112. In an embodiment, block 116 is configured to normalize the output of MHAB 114 and supply the normalized output to the next layer of encoder 106. For example, block 116 may include a normalization layer configured to normalize the final attention score matrix of MHAB 114 and supply the normalized matrix to the next MHAB of encoder 106. It should be noted that encoder 106 may include more than two MHABs, but only two are shown herein for illustrative purposes.

[0042] Block 118 represents a processing block configured to form the output of encoder 106. For example, block 118 may receive a final attention score matrix from a previous MHAB of the network and normalize the final attention score matrix of the MHAB to generate the output of encoder 106. In an embodiment, the output of encoder 106 is fed to the next layer of transformer network 105, which is configured to form the output for transformer network 105. For example, if transformer network 105 is configured to perform image classification, block 118 may feed its output to a multilayer perceptron (MLP) network configured to classify input images. Alternatively, if transformer network 105 is configured to perform object detection, block 118 may feed its output to an object detection network configured to output alarms when an object is detected, or when multiple distinct objects are detected.

[0043] Control logic 120 represents software executed by processing circuitry system 103 to manage the execution of encoder 106. For example, processing circuitry system 103 may execute control logic 120 to cause encoder 106 to perform a multi-head attention mechanism for performing tasks of converter network 105.

[0044] Figure 1B The layers of MHAB 110 in the example implementation are described. The layers of MHAB 110 represent processing layers configured to determine the scaled dot product attention of an image matrix through a series of fixed-point computations. In this implementation, MHAB 110 is configured to offload the fixed-point computations of its processing layers to an associated hardware accelerator. For example, processing circuitry system 103 may be coupled to a hardware accelerator configured to perform various fixed-point computations of operating environment 100. MHAB 110 includes, but is not limited to, an example matrix multiplication layer 119, an example SoftMax layer 121, and an example matrix multiplication layer 123. It should be noted that... Figure 1B The layers of MHAB 114 will be further explained, but for illustrative purposes, this paper will only discuss the layers of MHAB 110.

[0045] Matrix multiplication layer 119 represents the first processing layer of MHAB 110. The input to matrix multiplication layer 119 contains key data 115 and query data 117 of associated image matrices, while the output contains a first result matrix. The first result matrix represents a matrix storing attention scores of the associated image matrices. The attention scores represent the data of relevance assigned to the associated image matrices compared to other image matrices of the input image.

[0046] In one implementation, to perform matrix multiplication operations in matrix multiplication layer 119, processing circuitry 103 is configured to instruct an associated hardware accelerator to perform the operations. For example, processing circuitry 103 may instruct the hardware accelerator to perform matrix multiplication operations with respect to key data 115 and query data 117 of an associated image matrix. In response, the hardware accelerator is configured to read key data 115 from memory and write key data 115 to the left matrix input of the matrix multiplication operation, and to read query data 117 transposed from memory and write the transposed query data to the right matrix input of the matrix multiplication operation. Once written, the hardware accelerator is configured to produce a first result matrix by multiplying the left matrix input by the right matrix input.

[0047] In one implementation, matrix multiplication layer 119 is configured to perform matrix multiplication on each image matrix of the input image. For example, if the input image is embedded in four image matrices, matrix multiplication layer 119 is configured to cause the hardware accelerator to generate four first result matrices, such that each first result matrix corresponds to one of the four image matrices of the input image. In another implementation, matrix multiplication layer 119 is configured to perform matrix multiplication on each input matrix supplied to matrix multiplication layer 119. For example, if matrix multiplication layer 119 is supplied with six input matrices from the preceding layer (e.g., MHAB) of encoder 106, matrix multiplication layer 119 is configured to cause the hardware accelerator to generate six corresponding result matrices. Once generated, matrix multiplication layer 119 is configured to supply its output to SoftMax layer 121.

[0048] SoftMax layer 121 represents the second processing layer of MHAB 110. The input to SoftMax layer 121 contains the first result matrix, while the output contains the result of the SoftMax operation. The SoftMax operation represents a fixed-point computation used to normalize the attention scores generated by matrix multiplication layer 119. This means that the output of the SoftMax operation represents a second result matrix storing the normalized attention scores of the first image matrix. It should be noted that some transformer networks employ operations other than SoftMax to normalize the attention scores of the first matrix multiplication operation. Such examples can be found in the following publications: “SimA: Simple SoftMax-free Attention for Vision Transformers” by Soroush Koohpayegani et al., “SofterMax: Hardware / Software Co-Design of an Efficient SoftMax for Transformers” by Jacob Stevens et al., and “Replacing SoftMax with ReLU in Vision Transformers” by Mitchell Wortsman et al., which are hereby incorporated in their entirety.

[0049] In one implementation, to perform the SoftMax operation of SoftMax layer 121, the processing circuitry 103 is configured to instruct the associated hardware accelerator to perform fixed-point computation of the SoftMax operation. For example, the processing circuitry 103 may instruct the hardware accelerator to perform a height-direction SoftMax operation with respect to a first result matrix of the associated image matrix. In response, the hardware accelerator may generate a second result matrix for the associated image matrix. In another implementation, after generating the second result matrix, the hardware accelerator is configured to transpose and write the second result matrix to memory. For example, after performing the SoftMax operation of SoftMax layer 121, the hardware accelerator may transpose and write the result of the SoftMax operation to the associated memory.

[0050] In the implementation, SoftMax layer 121 is configured to perform a SoftMax operation on each output of matrix multiplication layer 119. For example, if matrix multiplication layer 119 outputs four first result matrices, SoftMax layer 121 is configured to cause the hardware accelerator to generate four second result matrices. Once generated, SoftMax layer 121 is configured to feed its output to matrix multiplication layer 123.

[0051] Matrix multiplication layer 123 represents the third processing layer of MHAB 110. The input of matrix multiplication layer 123 contains the value data 113 of the second result matrix written in transpose and the associated image matrix, while the output contains the third result matrix. The third result matrix represents the matrix storing the final attention score of the associated image matrix.

[0052] In an implementation, to perform matrix multiplication operations in matrix multiplication layer 123, processing circuitry 103 is configured to instruct an associated hardware accelerator to perform the operation. For example, processing circuitry 103 may instruct the hardware accelerator to perform matrix multiplication with respect to value data 113 of a transposed, written second result matrix and an associated image matrix. In response, the hardware accelerator is configured to read the transposed, written second result matrix from memory and write it to the left matrix input of the matrix multiplication operation, and to read the value data 113 from memory and write it to the right matrix input of the matrix multiplication operation. Once written, the hardware accelerator is configured to produce a third result matrix by multiplying the left matrix input by the right matrix input.

[0053] In the implementation, matrix multiplication layer 123 is configured to perform matrix multiplication on each output of SoftMax layer 121. For example, if SoftMax layer 121 outputs four second result matrices, matrix multiplication layer 123 is configured to cause the hardware accelerator to generate four third result matrices. Once generated, matrix multiplication layer 123 is configured to feed its output to the next layer of converter network 105. For example, matrix multiplication layer 123 may feed the third result matrices to a layer configured to generate a fourth result matrix by summing the data of the third result matrices together.

[0054] Figure 2 This describes an example method 200 for executing a converter network. Method 200 can be implemented in the context of software or program instructions that, when executed by a suitable computing system, direct the processing circuitry of the computing system to operate as follows, with reference incidentally. Figure 2 The steps in the process. For illustrative purposes, the following will be used. Figure 1A and 1BThe components are used to explain method 200. This is not intended to limit the application of scheduling method 200, but rather to provide an example.

[0055] First, block 108 generates embedded data (corresponding to) based on the sensor data collected by sensor 101. Figure 2 (Block 201). For example, block 108 may receive image data from sensor 101, divide the image data into multiple image blocks, embed these blocks into an equal number of image matrices, and supply the image matrices as input to MHAB 110. In response, MHAB 110 generates key data 115, query data 117, and value data 113 (corresponding to...) for each input matrix. Figure 2 (Block 203). For example, MHAB 110 can apply key weights, query weights, and value weights to each embedded block to generate key data 115, query data 117, and value data 113 for each embedded block.

[0056] Next, MHAB 110 is configured to perform matrix multiplication layer 119 (corresponding to...) Figure 2 (Block 205). In an embodiment, matrix multiplication layer 119 is performed by an associated hardware accelerator. For example, the hardware accelerator may be configured to read key data 115 of the first embedded block from memory and write the key data 115 to the left matrix input of the matrix multiplication operation. The hardware accelerator may be further configured to transpose and read query data 117 of the first embedded block from memory and write the transposed query data 117 to the right matrix input of the matrix multiplication operation. Finally, the hardware accelerator may be configured to produce a first result by performing matrix multiplication operations with respect to the left and right matrix inputs.

[0057] The first result represents a matrix storing the attention scores for the corresponding embedding block. In an implementation, the hardware accelerator is configured to generate the first result for each embedding block received by MHAB 110. For example, if MHAB 110 receives six different embedding blocks, the hardware accelerator is configured to generate a first result matrix for each of the six embedding blocks.

[0058] Next, matrix multiplication layer 119 outputs the first result to memory, and in response, MHAB 110 is configured to execute SoftMax layer 121 (corresponding to...). Figure 2(Block 207). In the implementation, the SoftMax layer 121 is executed by an associated hardware accelerator. For example, the associated hardware accelerator may be configured to read a first result from memory and perform a height-direction SoftMax operation on each of the first results to generate a set of second results. The set of second results represents storing each of the first results, and more specifically, a matrix of normalized attention scores for each embedded block.

[0059] In the implementation, the associated hardware accelerator is configured to transpose the second result and write it to memory. For example, if the output of the SoftMax layer contains six different second results, the hardware accelerator is configured to transpose each of the six different second results and write it to memory. Once stored in memory, MHAB 110 is triggered to execute matrix multiplication layer 123 (corresponding to...). Figure 2 (Box 209).

[0060] In this implementation, matrix multiplication layer 123 is performed by an associated hardware accelerator. For example, the hardware accelerator may be configured to read a transposed second result of an embedded block from memory and write the transposed second result to the left matrix input of the matrix multiplication operation. The hardware accelerator may be further configured to read value data 113 of a first embedded block from memory and write value data 113 to the right matrix input of the matrix multiplication operation. Finally, the hardware accelerator is configured to produce a third result by performing matrix multiplication operations with respect to the left and right matrix inputs.

[0061] The third result represents a matrix storing the final attention scores for the corresponding embedding block. In an implementation, the hardware accelerator is configured to generate the third result for each of the embedding blocks. For example, if the MHAB 110 receives six different embedding blocks, the hardware accelerator is configured to generate a third result matrix for each of the six embedding blocks.

[0062] Once generated, matrix multiplication layer 123 is configured to feed the generated third result to the next layer of transformer network 105. For example, matrix multiplication layer 123 may feed the third result to a layer configured to sum the data of the third result to generate a fourth result. The fourth result represents a matrix storing the final attention score for each of the embedded blocks. In one embodiment, the fourth result is fed to block 112.

[0063] Advantageously, method 200 utilizes the transpose read and transpose write capabilities of the hardware accelerator, thereby improving the efficiency of the converter network. Furthermore, method 200 supplies key data 115 as the left matrix input to the first matrix multiplication operation and supplies the transposed read query data as the right matrix input to the first matrix multiplication operation, thereby allowing the hardware accelerator to perform SoftMax operations in the height direction instead of SoftMax operations in the width direction. Therefore, method 200 provides a technique for efficiently executing layers of the converter encoder, thereby optimizing the execution of the converter network.

[0064] The SoftMax operation in the height direction is more efficient than the SoftMax operation in the width direction. SoftMax is an operation that treats [h×K×K] (3×197×197 in this example) input data as a series of independent h×K vectors, each of length K. Each of these vectors must be SoftMaxed and produce a vector of the same length as output. SoftMax involves finding a maximum value within the vector for numerical stability and therefore involves intra-vector operations that are less suitable for Single Instruction Multiple Data (SIMD) architectures. The SoftMax operation in the height direction involves performing SoftMax on a set of vectors rather than on a single vector at a time. This can be maintained without any overhead from the producer of this data. SoftMax has several intermediate steps, and SoftMax allows the final output to be in the original layout (h×K×K) output without any additional cost. SoftMax can occur on a series of vectors, thus preventing the need for intra-vector operations. SoftMax can occur on h×K vectors, allowing for a large number of vectors and allowing better utilization of architectures with larger SIMD widths.

[0065] Now let's move on to the next attached image. Figure 3A The illustration illustrates an example system 300 configured to perform image classification using a transformer network. For example, system 300 may represent... Figure 1A Transformer network 105. System 300 includes, but is not limited to, example image 301, example linear projection circuit system 302, example transformer encoder 304, and example multilayer perceptron (MLP) network 306.

[0066] Image 301 represents input data for a converter network. For example, system 300 may be coupled to a camera configured to collect image data of the environment. In an embodiment, image 301 represents image data collected by a vehicle. For example, the vehicle may include multiple cameras configured to collect image data of the surrounding environment (e.g., other vehicles, pedestrians, etc.) and supply the image data to system 300. In response, system 300 is configured to divide image 301 into multiple blocks, represented herein by example image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319. Image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319 represent segments of image data corresponding to image 301. In the implementation scheme, image blocks 303, 305, 307, 309, 311, 313, 315, 317 and 319 are provided as inputs to the linear projection circuit system 302.

[0067] Linear projection circuitry system 302 represents a circuitry configured to embed image data into a format available to the converter encoder. For example, linear projection circuitry system 302 may be configured to embed image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319 into a representation that can be fed to the converter encoder 304. In one embodiment, linear projection circuitry system 302 is configured to embed image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319 into an image matrix. In another embodiment, linear projection circuitry system 302 is configured to embed image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319 into an image vector. In any case, the output of the linear projection circuit system 302 includes example embedded blocks 323, 325, 327, 329, 331, 333, 335, 337 and 339.

[0068] Embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 represent blocks of embedded image data. For example, embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 can represent matrices that correspondingly store the embedded image data of image blocks 303, 305, 307, 309, 311, 313, 315, 317, and 319. For illustrative purposes, embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 represent image matrices. This is not intended to limit the application of the proposed technique, but rather to provide an example.

[0069] In the implementation scheme, before outputting the embedded blocks, the linear projection circuitry system 302 is configured to label the embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 with position embedding. For example, the linear projection circuitry system can sequentially label the embedded blocks such that embedded block 323 is labeled "1", embedded block 325 is labeled "2", and so on. Once labeled, the linear projection circuitry system 302 can provide the embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 as inputs to the converter encoder 304.

[0070] Transformer encoder 304 represents a deep learning architecture configured to employ an attention mechanism to perform tasks of system 300. For example, transformer encoder 304 may represent... Figure 1A The encoder 106. In the implementation, the converter encoder 304 employs a multi-head attention mechanism to perform image classification, as will be discussed later. Figure 3B Let's discuss this in detail.

[0071] The input to the transducer encoder 304 includes the output of the linear projection circuitry system 302 and an example classification embedding 321. The classification embedding 321 represents learnable data generated during the training phase of the system 300. For example, if the system 300 is trained to classify images within a car environment, the classification embedding 321 provides data that allows the transducer encoder 304 to classify vehicles, pedestrians, traffic lights, and other surrounding elements. In an embodiment, the linear projection circuitry system 302 is configured to label the classification embedding 321 with a location embedding. For example, the linear projection circuitry system may label the classification embedding as "0". It should be noted that the classification embedding 321 may represent alternative learnable embeddings (e.g., detection embeddings), but for illustrative purposes, the classification embedding 321 will be discussed herein.

[0072] In one implementation, the transducer encoder receives classification embedding 321 and embedding blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339, and in response, generates an attention-based output. For example, the transducer encoder 304 may generate a matrix storing the final attention score of image 301. The final attention score represents the relevance data assigned to the image data captured by image 301. The relevance of the image data describes the importance of the image data within the context of the task to which system 300 is configured to perform. In another implementation, after generating the final attention score matrix, the transducer encoder 304 is configured to provide its output to MLP network 306.

[0073] MLP network 306 represents a deep learning network configured to form the output of system 300. For example, MLP network 306 may include multiple layers configured to classify data of image 301. In an embodiment, MLP network 306 is configured to classify image 301 based on the output of transformer encoder 304. For example, MLP network 306 may classify image 301 as a car based on the final attention score matrix generated by transformer encoder 304.

[0074] Figure 3B This section describes an example layer of the transformer encoder 304 in the implementation scheme. The layers of the transformer encoder 304 represent processing layers configured to perform various attention-based operations. For example, the layer may perform operations for implementing multi-head attention mechanisms and scaled dot product attention mechanisms.

[0075] In one implementation, the converter encoder 304 is configured to offload the fixed-point computations of its processing layer to an associated hardware accelerator. For example, system 300 may be coupled to a hardware accelerator configured to perform various fixed-point computations of the converter network. The converter encoder 304 includes, but is not limited to, an example normalization layer 308, an example multi-head attention block (MHAB) 310, an example summation layer 312, an example normalization layer 314, an example multilayer perceptron (MLP) 316, and an example summation layer 318.

[0076] Normalization layer 308 represents a processing layer configured to generate input data for executing the multi-head attention mechanism of transformer encoder 304. For example, normalization layer 308 could represent... Figure 1A Block 108. In the implementation, normalization layer 308 is configured to normalize the data of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339, and feeds the normalized blocks to MHAB 310. In response, MHAB 310 is configured to apply various weight values ​​to the normalized blocks to generate key data, query data, and value data for embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339.

[0077] The query data for embedded blocks represents a matrix describing the viewpoint of the block within the input image. For example, the query data for embedded block 323 could indicate that embedded block 323 represents image block 303 of image 301. The key data for embedded blocks represents a matrix describing the relationship between the block and other blocks within the input image. For example, the key data for embedded block 323 could indicate that embedded block 323 includes image data corresponding to embedded blocks 305 and 309. The value data for embedded blocks represents a matrix describing the actual data of the block. For example, the value data for embedded block 323 could store the image data of image block 303.

[0078] MHAB 310 represents a processing block configured to perform a multi-head attention mechanism. For example, MHAB 310 could represent... Figure 1A MHAB 110 or MHAB 114. In an embodiment, MHAB 310 includes multiple processing layers configured to compute scaled dot product attention for each image matrix of the input image. For example, MHAB 310 may include a first matrix multiplication layer (e.g., matrix multiplication layer 119), a SoftMax layer (e.g., SoftMax layer 121), and a second matrix multiplication layer (e.g., matrix multiplication layer 123), which will be discussed later. Figure 3C Detailed discussion. The output of MHAB 310 is provided as input to the summing layer 312.

[0079] Summation layer 312 represents a processing layer configured to sum the output of MHAB 310 with the data in embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339. In an embodiment, the summation operation of summation layer 312 is performed by an associated hardware accelerator. For example, the associated hardware accelerator may sum the output of MHAB 310 with the data in embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339. The output of summation layer 312 is provided to normalization layer 314.

[0080] Normalization layer 314 represents a processing layer configured to normalize the output of summation layer 312. For example, normalization layer 314 normalizes the final attention score matrix of image 301. The output of normalization layer 314 is provided to MLP 316.

[0081] MLP 316 represents a processing block configured to linearize the output of normalization layer 314. For example, MLP 316 can linearize the final attention score matrix of image 301. This means that MLP 316 can linearly store the data of the final attention score matrix in memory. The output of MLP 316 is provided as input to summing layer 318.

[0082] Summation layer 318 represents a processing layer configured to sum the output of summation layer 312 with the output of MLP 316. For example, summation layer 318 may sum the final attention score matrix of image 301 with linearized data. In an embodiment, the summation operation of summation layer 318 is performed by an associated hardware accelerator. For example, the associated hardware accelerator may sum the output of summation layer 312 with data from the final attention score matrix. In an embodiment, the output of summation layer 318 is provided to MLP network 306. In another embodiment, the output of summation layer 318 is provided to the layer below encoder 304. For example, summation layer 318 may provide its output to a normalization layer configured to generate input data for performing another multi-head attention mechanism of encoder 304. It should be noted that encoder 304 may include multiple MHABs configured to determine scaled dot product attention of its inputs.

[0083] Additional example details of the layers that perform the transformer encoder in the context of the transformer network can be found in the following publication: entitled “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, written by Alexey Dosovitskiy et al.

[0084] Figure 3C This section describes an example layer of MHAB 310 in the implementation scheme. The layers of MHAB 310 represent processing layers configured to determine scaled dot product attention of an image matrix through a series of fixed-point computations. Scaled dot product attention represents an attention mechanism used to determine a normalized attention score for the input image. MHAB 310 includes, but is not limited to, example linearization layers 320, 322, and 324, example scaled dot product attention (SDPA) block 326, example cascaded layer 338, and example linearization layer 340.

[0085] Linearization layers 320, 322, and 324 respectively represent processing layers configured to linearize the key data, query data, and value data of embedded blocks in memory. For example, linearization layers 320, 322, and 324 may be configured to linearize the key data, query data, and value data of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 in memory respectively. In embodiments, linearization layers 320, 322, and 324 each include multiple processing layers such that the number of processing layers equals the number of embedded blocks supplied. For example, linearization layer 320 includes nine processing layers for linearizing the key data of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339. Similarly, linearization layers 322 and 324 comprise nine processing layers for linearizing the query data and value data of the embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339, respectively. In this implementation, the linearization operations of linearization layers 320, 322, and 324 are performed by an associated hardware accelerator. The outputs of linearization layers 320, 322, and 324 are supplied to SDPA block 326.

[0086] SDPA block 326 represents a processing block configured to determine the scaling dot product attention of the embedded data. For example, SDPA block 326 may be configured to determine the scaling dot product attention of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339. In an embodiment, the SDPA block includes multiple SDPA processing layers such that the number of SDPA processing layers equals the number of embedded blocks supplied. For example, SDPA block 326 may include nine processing layers for determining the scaling dot product attention of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339. In an embodiment, each SDPA processing layer of SDPA block 326 includes an example matrix multiplication layer 328, an example scaling layer 330, an example mask layer 332, an example SoftMax layer 334, and an example matrix multiplication layer 336.

[0087] Matrix multiplication layer 328 represents a processing layer configured to perform matrix multiplication operations on key data and query data embedded in the chunks. For example, matrix multiplication layer 328 could represent... Figure 1B The matrix multiplication layer 119. In an embodiment, the matrix multiplication operations of the matrix multiplication layer 328 are performed by an associated hardware accelerator. For example, system 300 may include a hardware accelerator configured to perform fixed-point computations of SDPA block 326.

[0088] In one implementation, to perform matrix multiplication operations in matrix multiplication layer 328, the hardware accelerator is configured to read linearized key data of the embedded blocks from memory and write the linearized key data to the left matrix input of the matrix multiplication operation. Next, the hardware accelerator is configured to transpose and read linearized query data of the embedded blocks from memory and write the transposed query data to the right matrix input of the matrix multiplication operation. Once written, the hardware accelerator is configured to produce a first result matrix by multiplying the left matrix input by the right matrix input. The first result matrix represents a matrix storing the attention scores of the embedded blocks (e.g., embedded block 323). In another implementation, matrix multiplication layer 328 is configured to feed the first result matrix to scaling layer 330.

[0089] Scaling layer 330 represents a processing layer configured to scale the output of matrix multiplication layer 328. For example, scaling layer 330 may be configured to format the data of a first result matrix into a representation more suitable for execution of SoftMax layer 334 by applying scaling values ​​to the first result matrix. In an embodiment, the scaling operation of scaling layer 330 is performed by an associated hardware accelerator. For example, the associated hardware accelerator may be configured to apply scaling values ​​to the first result matrix. The output of scaling layer 330 is supplied to mask layer 332 (or SoftMax layer 334).

[0090] Mask layer 332 represents an optional processing layer configured to mask the output of scaling layer 330. For example, mask layer 332 may be configured to format the output of scaling layer 330 into a representation more suitable for execution of SoftMax layer 334 by masking invalid values ​​of the scaled first result matrix. In embodiments, the masking operation of mask layer 332 is performed by an associated hardware accelerator. For example, the associated hardware accelerator may be configured to mask invalid data of the scaled first result matrix. The output of mask layer 332 is supplied to SoftMax layer 334. It should be noted that if SDPA block 326 does not contain mask layer 332, scaling layer 330 is configured to supply its output to SoftMax layer 334.

[0091] SoftMax layer 334 represents the processing layer configured to perform SoftMax operations. For example, SoftMax layer 334 could represent... Figure 1BThe SoftMax layer 121. In an embodiment, the SoftMax operation of the SoftMax layer 334 is performed by an associated hardware accelerator. For example, the associated hardware accelerator may be configured to perform a height-direction SoftMax operation relative to the output of the mask layer 332 (or scaling layer 330) to generate a second result matrix. The second result matrix represents a matrix storing the normalized attention scores of the first result matrix. In an embodiment, after generating the second result matrix, the hardware accelerator is configured to transpose the second result matrix and write it to memory. For example, after performing the SoftMax operation of the SoftMax layer 334, the associated hardware accelerator may transpose the second result matrix and write it to memory. Once written, the SoftMax layer 334 is configured to provide the transposed second result matrix as input to the matrix multiplication layer 336.

[0092] Matrix multiplication layer 336 represents a processing layer configured to perform matrix multiplication operations with respect to the transposed second result and value data of the embedded blocks. For example, matrix multiplication layer 336 could represent... Figure 1B The matrix multiplication layer 123. In an implementation, the matrix multiplication operations of the matrix multiplication layer 336 are performed by an associated hardware accelerator. For example, the associated hardware accelerator may be configured to read a transposed second result matrix from memory and write the transposed second result matrix to the left matrix input of the matrix multiplication operation. Next, the hardware accelerator may be configured to read embedded block value data from memory and write the value data to the right matrix input of the matrix multiplication operation. Once written, the hardware accelerator is configured to produce a third result matrix by multiplying the left matrix input by the right matrix input matrix.

[0093] The third result matrix represents the matrix storing the final attention scores of the embedding blocks. In the implementation, the third result matrix for each embedding block is fed as input to the cascade layer 338. For example, after SDPA block 326 generates the third result matrix for each of the embedding blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339, SDPA block 326 can feed each third result matrix to the cascade layer 338.

[0094] Cascade layer 338 represents a processing layer configured to cascade the outputs of SDPA block 326 into a singular matrix. For example, cascade layer 338 can cascade the third result matrices of embedded blocks 323, 325, 327, 329, 331, 333, 335, 337, and 339 into a singular matrix. In an embodiment, the cascading operation of cascade layer 338 is performed by an associated hardware accelerator. The output of cascade layer 338 is fed as input to linearization layer 340.

[0095] Linearization layer 340 represents a processing layer configured to linearize the output of cascaded layer 338. For example, linearization layer 340 may receive the output matrix of cascaded layer 338 and, in response, linearize the data of the output matrix in memory. In an embodiment, the linearization operation of linearization layer 340 is performed by an associated hardware accelerator. The output of linearization layer 340 is supplied to summing layer 312.

[0096] Figure 4 This is a block diagram of example environment 400, in which example model quantizer circuit system 405 operates to quantize example trained floating-point machine learning model 410 to generate corresponding example fixed-point machine learning model 415. Figure 4 The model quantizer circuit system 405 can be instantiated (e.g., created, generated, implemented, etc., over any time period) by a programmable circuit system, such as a central processing unit (CPU) that executes the first instruction. Alternatively, Figure 4 The model quantizer circuit system 405 can be instantiated (e.g., instantiated, generated, implemented, etc. over any time period) by (i) an application-specific integrated circuit (ASIC) or (ii) a field-programmable gate array (FPGA) that is constructed or configured to perform operations corresponding to the first instruction in response to the execution of the second instruction. Therefore, Figure 4 Some or all of the circuit systems can be instantiated at the same or different times. Figure 4 Some or all of the circuit system can be instantiated, for example, in one or more threads that execute in parallel or serially on hardware. Furthermore, in some examples, Figure 4 Some or all of the circuitry in the system may be implemented by a microprocessor circuitry that executes instructions or an FPGA circuitry that performs operations, to implement one or more virtual machines or containers.

[0097] exist Figure 4 In the illustrated example, environment 400 includes example workstation 420, example device configuration platform 425, and example target device 430. In the illustrated example, workstation 420 includes or otherwise implements model quantizer circuitry 405. Workstation 420 can be implemented by any computing device, processor platform, computer, server, etc. In some examples, workstation 420 is... Figure 14 An example implementation of the programmable circuit system platform 1400 is described below in further detail.

[0098] In the illustrated example, workstation 420 and, by extension, model quantizer circuitry 405 include example model input 435 to accept a trained floating-point model 410. In some examples, model input 435 may be implemented via a network interface, user input, etc., to accept a trained floating-point model 410 in the form of one or more data files, data structures, etc., specifying the structure of the various layers of model 410 and the values ​​of trained weights, biases, and / or other parameters for each layer of model 410. For example, the trained floating-point model 410 may correspond to a converter network, such as those described above. Figures 1A to 3C The described transformer network 105, and model input 435, can accept, retrieve, or otherwise obtain one or more data files, data structures, etc., which specify the structure of each layer of the transformer network 105 and the values ​​of the trained weights, biases, and / or other parameters of each layer of the transformer network 105. In some instances, the trained floating-point model 410 can correspond to any other type of machine learning model, such as neural networks, convolutional neural networks, reinforcement learning models, etc.

[0099] In the illustrated example, workstation 420, and by extension, model quantizer circuitry 405, also includes example precision input 440 to receive example fixed-point model precision data 445, which specifies the precision of the weights and activations in the resulting fixed-point machine learning model 415. In some examples, precision data 445 may specify the precision of the weights and activations in the various layers of the resulting fixed-point machine learning model 415 in the form of bits used to represent the weights and activations in a given layer of the fixed-point machine learning model 415. For example, precision data 445 may specify that for a given layer of the fixed-point machine learning model 415, the weights and activations of the layer will be represented using eight (8) bits (or some other number of bits). In some examples, precision data 445 may specify different numbers of bits to be used to represent the weights and activations of the layer for a given layer of the fixed-point machine learning model 415. For example, precision data 445 may specify that for a given layer of a fixed-point machine learning model 415, the weights of the layer will be represented by four (4) bits (or some other number of bits) and the activations of the layer will be represented by eight (8) bits (or some other number of bits different from the number of bits used to represent the weights).

[0100] In the illustrated example, workstation 420, and by extension, model quantizer circuitry 405, further includes example calibration data input 450 to receive example calibration data 455 used by model quantizer circuitry 405 to quantize the trained floating-point machine learning model 410 to generate a corresponding fixed-point machine learning model 415. In some examples, calibration data 455 includes input data elements and corresponding ground fact inference results expected to be processed and output by the trained floating-point machine learning model 410. For example, if the trained floating-point machine learning model 410 corresponds to the aforementioned transformer network 105 and is trained to perform image classification, the calibration data may include a set of input images formatted to be input to the trained floating-point machine learning model 410 and a set of corresponding ground fact inference classifications expected to be output by the trained floating-point machine learning model 410 for those input images.

[0101] As disclosed in further detail below, the model quantizer circuitry 405 processes the trained floating-point machine learning model 410, precision data 445, and calibration data 455 to output example quantization factors 460 for quantizing the weights and / or activations at various layers of the floating-point machine learning model 410. At least a portion of this processing may involve performing inference using the trained floating-point machine learning model 410. For example, and in further detail below, the quantization factor 460 may include scaling and offset factors used to quantize the weights and / or activations at various layers of the floating-point machine learning model 410 to determine the quantized weights and / or activations for the corresponding layer of the fixed-point machine learning model 415. In some examples, the model quantizer circuitry 405 also uses specific quantization factors 460 (e.g., scaling and offset factors) determined for the weights at various layers of the floating-point machine learning model 410 to output example quantized weights 465 for the corresponding layer of the fixed-point machine learning model 415.

[0102] Figure 5 Explanation by Figure 4The model quantizer circuitry 405 performs an example quantization operation 500. In the illustrated example, the model quantizer circuitry 405 observes a set of example floating-point weights 505 for a given layer of a floating-point machine learning model 410. For example, the floating-point weights 505 may be represented as 32-bit floating-point values. The model quantizer circuitry 405 obtains (e.g., observes) the set of floating-point weights 505 by causing the floating-point machine learning model 410 to perform and process at least a portion of calibration data 455 (e.g., perform inference on at least a portion of the calibration data 455). As disclosed in further detail below, the model quantizer circuitry 405 determines a quantization factor 460 for quantizing the set of floating-point weights 505, or in other words, converting the set of floating-point weights 505 into a corresponding set of quantized fixed-point weights 510. For example, the fixed-point weights 510 may be represented as 8-bit integer values.

[0103] Return to Figure 4 As illustrated in the examples, workstation 420 and, by extension, model quantizer circuitry 405 include an example quantization factor output 470 to output quantization factor 460 to device configuration platform 425. In some examples, workstation 420 and, by extension, model quantizer circuitry 405 also include an example quantized weight output 475 to output quantized weight 465 to device configuration platform 425. Device configuration platform 425 operates to download, install, or otherwise configure fixed-point machine learning model 415 on target device 430. Target device 430 may be any device capable of executing or otherwise implementing fixed-point machine learning model 415. For example, target device 430 may be a SoC device, an embedded processor device, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a computer, a smartphone, a tablet computer device, or any other computing device.

[0104] In the illustrated example, the device configuration platform 425 accepts a quantization factor 460, quantized weights 465, and an example fixed-point model structure 480. The fixed-point model structure 480 may take the form of one or more data files, data structures, etc., specifying the structure of the layers of a fixed-point machine learning model 415. In the illustrated example, the device configuration platform 425 uses the quantization factor 460 and the quantized weights 465 to configure the fixed-point model structure 480 for operation on the target device 430. For example, the device configuration platform 425 communicates with the target device 430 via an example configuration interface 485 to download, install, or otherwise configure the fixed-point model structure 480 on the target device 430. In some examples, the device configuration platform 425 also communicates with the target device 430 via a configuration interface 485 to populate the weights of the fixed-point model structure 480 on the target device 430 with the quantized weights 465. In some examples, the device configuration platform 425 also communicates with the target device 430 via a configuration interface 485 to populate the activation quantization operation of the fixed-point model structure 480 on the target device 430 with a quantization factor 460 (e.g., which may produce channel direction limiting values, etc., for the fixed-point machine learning model 415). In some examples, after configuring the fixed-point model structure 480 on the target device 430 based on the quantized weights 465 and the quantization factor 460, the device configuration platform 425 enables the target device 430 to execute the resulting populated fixed-point machine learning model 415.

[0105] Therefore, the device configuration platform 425 can be any platform capable of downloading, installing, or otherwise configuring the fixed-point machine learning model 415 on the target device 430. For example, the device configuration platform 425 can be implemented by a wireless transceiver, and the configuration interface 485 can be a wireless interface to allow over-the-air configuration of the target device 130. In some examples, the device configuration platform 425 is implemented by an electronic design automation (EDA) tool, and the configuration interface 485 can be a tool interface for communicating with the target device 130, such as a Joint Test Action Group (JTAG) interface. In some examples, the device configuration platform 425 is implemented by a computing device such as a computer, server, smartphone, etc., and the configuration interface 485 can be a communication interface for communicating with the target device 130, such as a serial port, Universal Serial Bus (USB), wireless interface, etc. In some examples, the device configuration platform 425 is contained in or otherwise implemented by a workstation 420. In some instances, the device configuration platform 425 is included in or otherwise implemented by the model quantizer circuit system 405.

[0106] Figure 4The example model quantizer circuit system 405 includes an example model observation circuit system 492, an example observation limiting circuit system 494, and an example model parameter quantization circuit system 496 to process a trained floating-point machine learning model 410, precision data 445, and calibration data 455 to generate a quantization factor 460, which will be used to quantize the weights and / or activations at various layers of the floating-point machine learning model 410 to generate a fixed-point machine learning model 415. In some examples, the model observation circuit system 492, the observation limiting circuit system 494, and the model parameter quantization circuit system 496 implement a post-trained quantization (PTQ) algorithm enhanced to support data observation limiting, as described in further detail below. For example, Figure 6 The description is based on the model quantizer circuit system 405 and more specifically the model parameter quantization circuit system 496. Figure 4 The example quantization factor 600 is determined from the observation data obtained by the model observation circuit system 492 and the observation value limiting circuit system 494.

[0107] Generally, for a given set of floating-point model parameters to be quantized, the model quantizer circuit system 405 determines a corresponding set of quantization factors 600. For example, for a given set of floating-point weights at a given layer of a trained floating-point machine learning model 410, the model quantizer circuit system 405 determines a corresponding set of quantization factors 600, which are used to quantize the set of floating-point weights to determine a corresponding set of floating-point weights for the layer of a fixed-point machine learning model 415. Similarly, for a given set of floating-point activations at a given layer of a trained floating-point machine learning model 410, the model quantizer circuit system 405 determines a different set of quantization factors 600, which are used to quantize the set of floating-point activations to determine a corresponding set of fixed-point weights for the layer of a fixed-point machine learning model 415. Therefore, for a given layer of a trained floating-point machine learning model 410, the model quantizer circuit system 405 can determine a first set of quantization factors 600 for the weights of the layer and a second set of quantization factors 600 for the activations of the layer.

[0108] like Figure 6 As shown, in some examples, a given set of quantization factors 600 includes an example scaling factor 605. In some examples, a given set of quantization factors 600 also includes an example offset factor 610. The set of quantization factors 600 is used to configure a quantizer function that performs a linear mapping to convert floating-point values ​​in the input range (α, β) to an output quantization range (α, β). q ,β q The input is a fixed-point or integer value, where α corresponds to the minimum possible input value and β corresponds to the maximum possible input value. qCorresponding to the minimum possible output value, and β q This corresponds to the maximum possible output value. Quantization typically reduces the processor and memory requirements of machine learning models (e.g., transformer networks and other types of neural networks) by decreasing the precision of their weights and activations.

[0109] Figure 6 The illustration describes two example quantizer functions 615 and 620 that can be configured by the set of quantization factors 600. Quantizer function 615 is an example of a symmetric quantizer that maps a symmetric floating-point input range (α, β) to a symmetric fixed-point or integer output range (α, β). q ,β q ), where α = -β and α q =-β q Quantizer function 620 is an example of an asymmetric quantizer that maps an asymmetric floating-point input range (α, β) to an asymmetric fixed-point or integer output range (α, β). q ,β q ), where α ≠ -β and α q ≠-β q .

[0110] like Figure 6 As illustrated in the example, the model parameter quantization circuit system 496 is based on the ratio of Equation 1, which is based on the floating-point input range (α, β) and the fixed-point or integer output range (α). q ,β q Calculate the scaling factor 605:

[0111]

[0112] In Equation 1, the fixed-point or integer output range β q -α q The size is based on the precision of fixed-point or integer values, as specified by the fixed-point model precision data 445 applied to precision input 440. If the precision for a particular set of fixed-point or integer values ​​is specified as b bits by precision data 445, then the fixed-point or integer output quantization range is β. q -α q The size is given by Equation 2:

[0113] β q -α q =2 b -1

[0114] Equation 2

[0115] For example, if b is set to 8-bit precision, the quantized output range is given by Equation 3:

[0116] β q -αq =2 8 -1 = 255

[0117] Equation 3

[0118] In some instances, the model parameter quantization circuit system 496 is based on Equation 3, using the floating-point input range (α, β) and the fixed-point or integer output range (α). q ,β q Calculate the offset factor 610:

[0119]

[0120] As shown in Equations 1 and 4 above, for a given set of floating-point values, the quantization factor 600, and more specifically the scaling factor 605 and the offset factor 610, depend on the size of the floating-point input range β-α, which depends on the knowledge of the minimum possible input value α and the maximum possible input value β. Return to Figure 4 The model observation circuitry 492 of the model quantizer circuitry 405 obtains a trained floating-point machine learning model 410 via model input 435. The model observation circuitry 492 then inserts observer operations (also referred to as observer functions, observers, etc.) into the trained floating-point machine learning model 410 to observe the floating-point weights and activations at various layers of the model 410. In the illustrated example, the model observation circuitry 492 also obtains at least a portion of calibration data 455 via calibration data input 450 and causes the trained floating-point machine learning model 410 to process the calibration data 455 (e.g., perform inference on the calibration data 455). While the model 410 processes the input calibration data 455 (e.g., perform inference on the input calibration data 455), the model observation circuitry 492 then uses the inserted observer operations to collect observations of the floating-point weights and activations at various layers of the trained floating-point machine learning model 410.

[0121] Figure 7 Explanation by Figure 4 The model observation circuit system 492 of the model quantizer circuit system 405 observes example activation data 700 at the activation outputs of various example layers 705 of the trained floating-point machine learning model 410. Model layer 705 corresponds to... Figure 3B The converter encoder 304 includes summation layer 312, MLP layer 316, and summation layer 318. Figure 7 In the illustrated example, the model observation circuit system 492 inserts observer operations at the active outputs of summing layer 312, MLP layer 316, and summing layer 318 to observe the corresponding example active output data 712, 716, and 718 at those respective layers.

[0122] Using the observed activation data 712, the minimum and maximum values ​​of data 712 (e.g., α1, β1) can be determined, and a first set of quantization factors 600 (e.g., corresponding to the first scaling factor S1 and the first offset factor Z1) can be calculated using Equations 1 and 4 above. These quantization factors can be used to quantize the activation at the summation layer 312. Similarly, using the observed activation data 716, the minimum and maximum values ​​of data 716 (e.g., α2, β2) can be determined, and a second set of quantization factors 600 (e.g., corresponding to the second scaling factor S2 and the second offset factor Z2) can be calculated using Equations 1 and 4 above. These quantization factors can be used to quantize the activation at the MLP layer 316. Likewise, using the observed activation data 718, the minimum and maximum values ​​of data 718 (e.g., α3, β3) can be determined, and a third set of quantization factors 600 (e.g., corresponding to the third scaling factor S3 and the third offset factor Z3) can be calculated using Equations 1 and 4 above. These quantization factors can be used to quantize the activation at the summation layer 318.

[0123] However, in Figure 7 In the illustrated examples, the observed activation data 712, 716, and 718 contain example outliers 722, 726, and 728, respectively, which increase the corresponding observation range of the floating-point data to be quantized, thereby reducing quantization precision. For example, representing the smaller input 32-bit floating-point range of [0,1] with an output 8-bit integer range of [0,255] is more accurate than representing the larger input 32-bit floating-point range of [0,1000] with the same output 8-bit integer range of [0,255]. Furthermore, the presence of such outliers also adversely affects the precision of smaller quantized values, potentially leading to incorrect predictions.

[0124] therefore, Figure 4 The model quantizer circuit system 405 includes an observation-limiting circuit system 494 to limit outliers in the observation data obtained by the model observation circuit system 492 to improve quantization accuracy. For example, the observation-limiting circuit system 494 can limit outliers 722, 726, and 728 from observed active data 712, 716, and 718, reducing the overall range of the active data 712, 716, and 718 to be quantized. In some examples, to perform this limiting, the observation-limiting circuit system 494 determines one or more limiting thresholds based on the observed floating-point values. For example, the observation-limiting circuit system 494 may determine one or more example limiting thresholds 732 based on observed active data 712. Similarly, the observation-limiting circuit system 494 may determine one or more example limiting thresholds 736 based on observed active data 716, and one or more example limiting thresholds 738 based on observed active data 718.

[0125] For example, to determine the limiting threshold 732, the observation limiting circuit system 494 can use the observed activation data 712 to determine one or more measures. For example, the measures determined by the observation limiting circuit system 494 from the observed activation data 712 may include standard deviation, variance or variance-based measures, distribution-based measures, deviation measures, skewness measures, percentile measures, etc. In some such instances, the observation limiting circuit system 494 then uses the measures to limit the observed activation data 712. In some such examples, the observation limiting circuit system 494 may scale the calculated measures by a number to determine scaled measures, and then use said scaled measures to limit the observed activation data 712. Similarly, the observation limiting circuit system 494 can use the observed activation data 716 and 718 to determine corresponding measures, and use those measures to limit the observed activation data 716 and 718. Figure 8 Explanation by Figure 4 The example function 800 is used by the model quantizer circuit system 405 and the observation limiting circuit system 494 to perform outlier removal to quantize the trained floating-point machine learning model.

[0126] Turn Figure 8 In order to use function 800 to determine the limiting threshold 732, for the observed activation data 712, the observed value limiting circuit system 494 determines the standard deviation of the observed activation data 712 as (by... Figure 8 x in std The first metric is (represented by). The observation limiting circuit system 494 also determines the mean or average value (by) the observed activation data 712. Figure 8 x in mean The second metric is the mean and standard deviation. The observation limiting circuit system 494 then uses the mean and standard deviation metrics to determine the limiting threshold 732, which in the illustrated example includes an example upper limiting threshold 805 and an example lower limiting threshold 810. For example, the observation limiting circuit system 494 determines the upper limiting threshold 805 as the standard deviation multiplied by a number (e.g., ...). Figure 8 The result is the sum of the number "3" in the mean (or some other number such as 2 or 4) and the mean. In this example, the observation limiting circuit system 494 determines the lower limit threshold 810 as the result of subtracting the standard deviation multiplied by a number (e.g., ...) from the mean. Figure 8 The result is the number "3" (or some other number). In some examples, using the number "3" to scale the standard deviation is referred to as the three-sigma method for determining the clipping threshold. In some examples, the observation clipping circuit system 494 uses function 800 to perform a similar operation to determine the corresponding clipping thresholds 736 and 738 for the observed active data 716 and 718.

[0127] Using function 800, the observation limiting circuit system 494 then limits the observed activation data 712 by a value greater than the upper limit threshold 805 (by...). Figure 8 The "x" in the equation represents the value of the observed activation data 712. The observed activation data 712 is set to be equal to the upper limit threshold 805 to limit those values. Similarly, using function 800, the observation limiting circuit system 494 limits those values ​​by setting the observed activation data 712 values ​​greater than the lower limit threshold 810 to be equal to the lower limit threshold 810. Furthermore, the observation limiting circuit system 494 leaves the observed activation data 712 values ​​between the upper limit threshold 805 and the lower limit threshold 810 unchanged. If all values ​​of the observed activation data 712 are between or within thresholds 805 and 810, the observation limiting circuit system 494 can be configured not to limit any values ​​of the observed activation data 712. The model parameter quantization circuit system 496 then uses the resulting limiting value (determined by the observed activation data) to limit the values ​​of the observed activation data. Figure 8 The "x" in new The quantization factor 600 (e.g., scaling factor 605 and offset factor 610) is used to determine the quantization factor 600 that will be used to quantize the activation data of the corresponding layer 312 in the fixed-point machine learning model 415. In some examples, the observation clipping circuitry 494 uses function 800 to perform a similar operation to clip the observed activation data 716 and 718 based on corresponding clipping thresholds 736 and 738. The model parameter quantization circuitry 496 then uses the resulting clipping value to determine the quantization factor 600 that will be used to quantize the activation data of the corresponding layers 316 and 318 in the fixed-point machine learning model 415.

[0128] Although the model observation circuit system 492, observation value limiting circuit system 494, and model parameter quantization circuit system 496 have been described from the perspective of quantized activation data of a given layer of the machine learning model 410, these circuit systems can also be used to quantize the weights of a given layer in a similar manner. Therefore, in summary, the model observation circuit system 492 of the model quantizer circuit system 405 receives the trained floating-point machine learning model 410 and calibration data 455 as input. The model observation circuit system 492 also inserts an observer operation to observe the values ​​of activations and weights at various layers of the trained floating-point machine learning model 410 as it processes the calibration data. The observation value limiting circuit system 494 of the model quantizer circuit system 405 uses a metric determined for the observed activation data and a metric determined for the observed weights to limit the observed activation data and observed weights at various layers of the trained floating-point machine learning model 410. For example, the observation value limiting circuit system 494 can use... Figure 8The function 800 and a metric determined for the observed activation data are used to limit the observed activation data for a given model layer. Similarly, the observation limiting circuit system 494 can use... Figure 8 The observed weights of a given model layer are clipped using a function 800 and a metric determined for the observed weight data. In some examples, when the trained floating-point machine learning model 410 processes calibration data 455, the observation clipping circuitry 494 also propagates clipped activation data at the output of a given layer to the next model layer (e.g., instead of propagating unclipped data). By preventing the propagation of outlier activations, the observation clipping circuitry 494 may be able to reduce the prevalence of outliers in subsequent model layers.

[0129] Next, the model parameter quantization circuitry 496 uses the obtained clipped activation data of a given model layer to determine a quantization factor 600 (e.g., scaling factor 605 and offset factor 610) to be used for quantizing the activation data of the corresponding layer in the fixed-point machine learning model 415. The model parameter quantization circuitry 496 also includes the quantization factor 600 for the activation data of the given layer in a quantization factor 460 output via quantization factor output 470. Similarly, the model parameter quantization circuitry 496 uses the obtained clipped weight values ​​of a given model layer to determine a quantization factor 600 (e.g., scaling factor 605 and offset factor 610) to be used for quantizing the weights of the corresponding layer in the fixed-point machine learning model 415. In some examples, the model parameter quantization circuitry 496 also includes the quantization factor 600 for the weights of the given layer in a quantization factor 460 output via quantization factor output 470. In some examples, if the trained weights of a given model layer do not change during the processing of calibration data 455, the model parameter quantization circuitry 496 also configures an instance of quantizer function 615 or 620 using a quantization factor 600 of the weights (e.g., a scaling factor 605 and an offset factor 610). The model parameter quantization circuitry 496 then uses the configured function 615 or 620 to quantize the weights of the given model layer, outputting them via quantized weight output 475.

[0130] Figure 9 Explanation by Figure 4 The model quantizer circuitry 405 supports two example quantization types 900. Example quantization type 900 includes per-channel quantization 905 and per-tensor quantization 910. In some examples, the model quantizer circuitry 405 implements per-channel quantization 905 to independently quantize the weights of different channels for a given model layer. For example, the observation clipping circuitry 494 independently clips the weight sets of different channels, and the model parameter quantization circuitry 496 independently determines the individual quantization factors for different weight sets. Figure 9The illustrated example depicts per-channel quantization 905 performed for three (3) channels 915, 920, and 925 with corresponding sets of weights 930, 935, and 940. The model parameter quantization circuitry 496 determines a first set of quantization factors 945 for the first set of weights 930 associated with the first channel 915, a second set of quantization factors 950 for the second set of weights 935 associated with the second channel 920, and a third set of quantization factors 955 for the third set of weights 940 associated with the third channel 925.

[0131] In some examples, the model quantizer circuitry 405 performs per-tensor quantization 910 to collectively quantize the weights of all channels for a given model layer. For example, observation-limiting circuitry 494 together limits the set of weights for all channels, and model parameter quantization circuitry 496 determines a set of quantization factors to be applied to all weights of the layer. In some examples, the model quantizer circuitry 405 performs per-tensor quantization 910 to collectively quantize the activations of all channels for a given model layer. For example, observation-limiting circuitry 494 together limits the activations of all channels, and model parameter quantization circuitry 496 determines a set of quantization factors to be applied to all activations of the layer. Figure 9 The illustrated example depicts per-tensor quantization 910 performed for three (3) channels 960, 965, and 970 with corresponding sets of activations 975, 980, and 985. The model parameter quantization circuitry 496 determines a single quantization factor 990 for the three sets of activations.

[0132] In some examples, the trained floating-point machine learning model 410 is a transformer network with outliers limited to activations in a few specific layers and / or channels of the transformer network. For example, the outliers may be limited to the MLP branches of the transformer network. In some such examples, the observation limiting circuitry system 494 is configured to limit its limiting operation to those layers / channels.

[0133] Based on the foregoing description, in some examples, the observation limiting circuitry 494 of the model quantizer circuitry 405 limits the values ​​of activations associated with layers of a floating-point version of a machine learning model (e.g., a trained floating-point machine learning model 410) to determine the limited values ​​of the activations, where the activation values ​​are based on calibration data 455 applied to the floating-point version of the machine learning model. In some such examples, the model parameter quantization circuitry 496 of the model quantizer circuitry 405 uses the limited values ​​of the activations to determine a quantization factor (e.g., quantization factor 600) to quantize the activations associated with corresponding layers of a fixed-point version of a machine learning model (e.g., a fixed-point machine learning model 415). In some such examples, the device configuration platform 425 uses the quantization factor to configure a fixed-point version of the machine learning model on a target device 430.

[0134] In some examples, the model observation circuitry 492 initiates the execution of a floating-point version of the machine learning model using calibration data 455, and the observation limiting circuitry 494 causes the activated, limited values ​​to propagate to subsequent layers of the floating-point version of the machine learning model during execution.

[0135] In some examples, the activation is a first activation, and the model observation circuitry 492 observes values ​​of multiple activations associated with a layer of a floating-point version of the machine learning model, wherein the values ​​of the multiple activations are based on calibration data 455 applied to the floating-point version of the machine learning model, and the multiple activations include the first activation. In some examples, the multiple activations correspond to a single channel associated with a layer of a floating-point version of the machine learning model. In some such examples, the observation limiting circuitry 494 uses the values ​​of the activations to determine a metric, and the observation limiting circuitry 494 uses the metric to limit the values ​​of the first activation. In some such examples, the observation limiting circuitry 494 scales the metric to determine a scaled metric, and uses the scaled metric to limit the values ​​of the first activation. In some such examples, the metric is the standard deviation of the activation values. In some such examples, the observation limiting circuitry 494 also determines the mean of the activation values, and uses the mean and standard deviation multiplied by a number to limit the values ​​of the first activation.

[0136] In some examples, the activation is a first activation, the quantization factor 600 includes a scaling factor 605, and the observation limiting circuitry system 494 determines the scaling factor 605 by: (i) determining the observation range of a plurality of activations associated with a layer of a floating-point version of the machine learning model, wherein the observations of the plurality of activations are based on calibration data 455 applied to the floating-point version of the machine learning model, and the plurality of activations include a first activation; and (iii) determining the scaling factor 605 using the ratio of the observation range to the quantization range associated with the corresponding layer of a fixed-point version of the machine learning model. In some such examples, the quantization factor 605 also includes an offset factor 610, and the observation limiting circuitry system 494 determines the offset factor 610 using the ratio of a first observation (e.g., the minimum observation) to the scaling factor 605.

[0137] In some examples, quantization factor 600 is the first quantization factor, and model observation circuitry 492 observes the values ​​of a first set of weights associated with a layer of the floating-point version of the machine learning model, where the first set of weights corresponds to a single channel associated with the layer of the floating-point version of the machine learning model. In some such examples, observation limiting circuitry 494 uses a metric (e.g., standard deviation) to limit the values ​​of the first weights in the first set of weights to determine a limited value for the first weights, where the metric is based on the values ​​of the first set of weights. In some such examples, model parameter quantization circuitry 496 uses the limited value of the first weights to determine a second quantization factor to be used to obtain a second set of quantized weights associated with the corresponding layer of the fixed-point version of the machine learning model.

[0138] In some examples, a floating-point version of the machine learning model is a floating-point version of the transformer network, and a layer of the floating-point version of the machine learning model is a layer of the floating-point version of the transformer network. In some such examples, a layer of the floating-point version of the transformer network corresponds to one of the following: (i) the output layer of a multilayer perceptron, (ii) a first-element-wise additive layer coupled to the output layer of a multilayer perceptron, or (iii) a second-element-wise additive layer coupled to the first-element-wise additive layer.

[0139] In some examples, the model quantizer circuit system 405 includes means for observing the machine learning model. For example, the means for observation may be implemented by the model observation circuit system 492. In some examples, the model observation circuit system 492 may be, for example... Figure 14 The example programmable circuit system 1412 is instantiated as a programmable circuit system. For example, the model observation circuit system 492 can be instantiated by... Figure 15 An example microprocessor 1500 is instantiated, the example microprocessor executing machine-executable instructions, such as at least those defined by... Figure 10 Blocks 1005-1025 and Figure 11Those implemented in block 1145. In some examples, the model observation circuitry system 492 may be instantiated by a hardware logic circuitry system, which may be an ASIC, XPU, or [other system] constructed to perform operations corresponding to machine-readable instructions. Figure 16 The FPGA circuit system 1600 is implemented. Alternatively, the model observation circuit system 492 can be instantiated by any other combination of hardware, software, or firmware. For example, the model observation circuit system 492 can be implemented by at least one or more hardware circuits (e.g., processor circuit systems, discrete or integrated analog or digital circuit systems, FPGAs, ASICs, XPUs, comparators, operational amplifiers, logic circuits, etc.) configured or constructed to execute some or all of machine-readable instructions or to perform some or all of the operations corresponding to machine-readable instructions without executing software or firmware, but other configurations are equally suitable.

[0140] In some examples, the model quantizer circuit system 405 includes means for limiting the observed model values. For example, the means for limiting may be implemented by an observation limiting circuit system 494. In some examples, the observation limiting circuit system 494 may be, for example... Figure 14 The example programmable circuit system 1412 is instantiated as a programmable circuit system. For example, the observation limiting circuit system 494 can be instantiated by... Figure 15 An example microprocessor 1500 is instantiated, the example microprocessor executing machine-executable instructions, such as at least those defined by... Figure 10 Block 1030 and Figure 11 Those implemented in blocks 1110, 1115, 1125, 1130, and 1145. In some examples, the observation limiting circuitry system 494 may be instantiated by a hardware logic circuitry system, which may be an ASIC, XPU, or [other system] constructed to perform operations corresponding to machine-readable instructions. Figure 16 The FPGA circuit system 1600 is implemented. Alternatively, the observation limiting circuit system 494 can be instantiated by any other combination of hardware, software, or firmware. For example, the observation limiting circuit system 494 can be implemented by at least one or more hardware circuits (e.g., processor circuit systems, discrete or integrated analog or digital circuit systems, FPGAs, ASICs, XPUs, comparators, operational amplifiers, logic circuits, etc.) configured or constructed to execute some or all of machine-readable instructions or to perform some or all of the operations corresponding to machine-readable instructions without executing software or firmware, but other configurations are equally suitable.

[0141] In some examples, the model quantizer circuit system 405 includes means for quantizing model parameters (e.g., activations and weights). For example, the means for quantizing model parameters may be implemented by a model parameter quantization circuit system 496. In some examples, the model parameter quantization circuit system 496 may be, for example... Figure 14 The example programmable circuit system 1412 is instantiated as a programmable circuit system. For example, the model parameter quantization circuit system 496 can be instantiated by... Figure 15 An example microprocessor 1500 is instantiated, the example microprocessor executing machine-executable instructions, such as at least those defined by... Figure 10 Block 1030 and Figure 11 Those implemented in blocks 1120, 1135, and 1150. In some examples, the model parameter quantization circuitry system 496 may be instantiated by a hardware logic circuitry system, which may be an ASIC, XPU, or [other device] constructed to perform operations corresponding to machine-readable instructions. Figure 16 The FPGA circuit system 1600 is implemented. Alternatively, the model parameter quantization circuit system 496 can be instantiated by any other combination of hardware, software, or firmware. For example, the model parameter quantization circuit system 496 can be implemented by at least one or more hardware circuits (e.g., processor circuit systems, discrete or integrated analog or digital circuit systems, FPGAs, ASICs, XPUs, comparators, operational amplifiers, logic circuits, etc.) configured or constructed to execute some or all of machine-readable instructions or to perform some or all of the operations corresponding to machine-readable instructions without executing software or firmware, but other configurations are equally suitable.

[0142] In some examples, the model quantizer circuit system 405 includes means for configuring a fixed-point machine learning model on a target device. For example, the means for configuration may be implemented by a device configuration platform 425. In some examples, the device configuration platform 425 may be, for example... Figure 14 The example programmable circuit system 1412 is instantiated. For example, the device configuration platform 425 may be instantiated by... Figure 15 An example microprocessor 1500 is instantiated, the example microprocessor executing machine-executable instructions, such as at least those defined by... Figure 10 Block 1030 and Figure 11 Those implemented by block 1150. In some examples, the device configuration platform 425 may be instantiated by a hardware logic circuit system, which may be an ASIC, XPU, or [other system] constructed to perform operations corresponding to machine-readable instructions. Figure 16The FPGA circuitry system 1600 is implemented. Alternatively, the device configuration platform 425 can be instantiated by any other combination of hardware, software, or firmware. For example, the device configuration platform 425 can be implemented by at least one or more hardware circuits (e.g., processor circuitry systems, discrete or integrated analog or digital circuitry systems, FPGAs, ASICs, XPUs, comparators, operational amplifiers, logic circuits, etc.) configured or constructed to execute some or all of machine-readable instructions or to perform some or all of the operations corresponding to machine-readable instructions without executing software or firmware, but other configurations are equally suitable.

[0143] Figure 10 This is a flowchart representing example machine-readable instructions and / or example operations 1000, which can be implemented, instantiated, or executed by a programmable circuit system. Figure 4 At least one of the model quantizer circuit systems 405. Figure 10 Example machine-readable instructions and / or example operations 1000 begin at block 1005, where the model observation circuitry 492 of the model quantizer circuitry 405 accesses the trained floating-point machine learning model 410 to be quantized, as described above. At block 1010, the model observation circuitry 492 fuses modules of the trained floating-point machine learning model 410, which can be combined together without affecting the quantization of the trained floating-point machine learning model 410. At block 1015, the model observation circuitry 492 inserts observer operations and any other program code stubs into the trained floating-point machine learning model 410 to allow observation of the weights and activations of the individual layers of the trained floating-point machine learning model 410, as described above.

[0144] At block 1020, model observation circuitry 492 accesses calibration data 455, as described above. At block 1025, model observation circuitry 492 causes the trained floating-point machine learning model 410 to execute (e.g., on workstation 420) and process (e.g., perform inference on it) the calibration data 455 to obtain observations of the weights and activations of the various layers of the trained floating-point machine learning model 410, as described above. At block 1030, observation-limiting circuitry 494 and model parameter quantization circuitry 496 of model quantizer circuitry 405 use the observed weights and activations obtained at block 1025 to determine quantization factors 460 for quantizing the weights and activations of the various layers of the trained floating-point machine learning model 410, as described above. For example, at block 1025, observation-limiting circuitry 494 and model parameter quantization circuitry 496 may execute an enhanced PTQ process, which uses the observed weights and activations obtained at block 1025 with limited values. Example machine-readable instructions and / or example operations that can be used to perform the processing of block 1030 are in Figure 11 The explanation is described in detail below.

[0145] At block 1035, the model parameter quantization circuitry 496 outputs a set of quantization factors 460 for quantizing the corresponding weight sets and activation sets at each layer of the trained floating-point machine learning model 410, as described above. In some examples, the model parameter quantization circuitry 496 also uses quantization factors 600 for the corresponding weight sets at each layer of the trained floating-point machine learning model 410 to quantize those weights and outputs a set of quantized weights 465 for the corresponding layers of the fixed-point machine learning model 415, as described above. In some examples, at block 1035, the device configuration platform 425 configures the fixed-point machine learning model 415 on the target device 430 using the quantization factors 460 and quantized weights 465 for the various model layers, as described above. Then, the example machine-readable instructions and / or example operation 1000 conclude.

[0146] Figure 11 This is a flowchart representing example machine-readable instructions and / or example operations 1030, which may be implemented, instantiated, or executed by a programmable circuit system to implement the model quantizer circuit system 405. Figure 10 At least one of the processes executed at block 1030. Figure 11Example machine-readable instructions and / or example operations 1030 begin at block 1105, where the observation limiting circuitry 494 of the model quantizer circuitry 405 accesses the observations of the activations and weights of a given layer of the trained floating-point machine learning model 410, as described above. Also as described above, the observations of the activations and weights are based on calibration data 455 applied to the trained floating-point machine learning model 410 by the model observation circuitry 492 of the model quantizer circuitry 405.

[0147] At block 1110, the observation-limiting circuitry 494 uses the observed activation values ​​to determine one or more activation metrics to be used for limiting the observed activation values ​​associated with a given layer of the trained floating-point machine learning model 410, as described above. For example, the activation metrics could be the standard deviation and mean of the observed activation values, as described above. At block 1115, the observation-limiting circuitry 494 uses the activation metrics determined at block 1110 to limit one or more observed activation values ​​of a given model layer to determine the corresponding limited activation values ​​for the given model layer, as described above. At block 1120, the model parameter quantization circuitry 496 of the model quantizer circuitry 405 uses the limited activation values ​​of the given model layer to determine a first set of quantization factors to be used for quantizing the activations associated with the corresponding layer of the fixed-point machine learning model 415, as described above.

[0148] At block 1125, the observation-limiting circuitry 494 uses the observed weight values ​​to determine one or more weight measures to be used for limiting the observed weight values ​​associated with a given layer of the trained floating-point machine learning model 410, as described above. For example, the weight measures could be the standard deviation and mean of the observed weight values, as described above. At block 1130, the observation-limiting circuitry 494 uses the weight measures determined at block 1125 to limit one or more observed weight values ​​of a given model layer to determine the corresponding limited weight values ​​for the given model layer, as described above. At block 1135, the model parameter quantization circuitry 496 uses the limited weight values ​​of the given model layer to determine a second set of quantization factors to be used for quantizing the weights associated with the corresponding layer of the fixed-point machine learning model 415, as described above.

[0149] At block 1140, the model quantizer circuitry 405 determines whether there is a subsequent layer of the trained floating-point machine learning model 410 to be quantized. If there is a subsequent model layer to be quantized (corresponding to the "yes" output of block 1140), the model observation circuitry 492 and / or the observation clipping circuitry 494 propagate the observed activation values ​​(including any clipped activation values) of the current layer to the next layer of the trained floating-point machine learning model 410, as described above. Processing then returns to block 1105 and the blocks following block 1105 to allow the weights and activations of the next model layer to be quantized. However, if there is no longer a model layer to be quantized (corresponding to the "no" output of block 1140), at block 1150, the model parameter quantization circuitry 496 outputs a set of quantization factors determined for each layer of the trained floating-point machine learning model 410, and causes the device configuration platform 425 to use the set of quantization factors to configure the fixed-point machine learning model 415 on the target device 430, as described above. Then, the machine-readable instructions and / or example operation 1030 end.

[0150] Figure 12 This indicates that, in the context of activating outlier removal, Figure 4 Example model quantization performance results 1200 are provided by the model quantizer circuit system 405. Results 1200 show that, compared to other model quantization methods that do not employ outlier limiting / suppression, the outlier limiting (also known as outlier suppression) performed on the activation data by the model quantizer circuit system 405 increases the accuracy of the quantized model and reduces the error of the quantized model.

[0151] Figure 13 illustrate Figure 4 The model quantizer circuit system 405 offers several advantages over other model quantization methods, as illustrated in Example 1300. Advantage 1300 includes (i) avoiding the use of mixed precision and the associated increase in model size and complexity, (ii) not involving changes to the structure of the quantized machine learning model, and (iii) not involving retraining of the machine learning model.

[0152] Figure 14 This is a block diagram of an example programmable circuit system platform 1400, which is configured to perform one or a combination of the following: implementation or instantiation. Figures 10 to 11 One or more of the example machine-readable instructions or example operations are used to implement Figure 4 The model quantizer circuit system 405. The programmable circuit system platform 1400 can be used for, for example, servers, personal computers, workstations, self-learning machines (e.g., neural networks), mobile devices (e.g., mobile phones, smartphones, e.g., iPads). TMTablet computers, personal digital assistants (PDAs), internet devices, DVD players, CD players, digital video recorders, Blu-ray players, game consoles, personal video recorders, set-top boxes, headphones (e.g., augmented reality (AR) headphones, virtual reality (VR) headphones, etc.) or other wearable devices, or any other type of computing or electronic device.

[0153] The illustrated example programmable circuit system platform 1400 includes a programmable circuit system 1412. The illustrated example programmable circuit system 1412 is hardware. For example, the programmable circuit system 1412 may be implemented by one or more integrated circuits, logic circuits, FPGAs, microprocessors, CPUs, GPUs, DSPs, or microcontrollers from any desired family or manufacturer. The programmable circuit system 1412 may be implemented by one or more semiconductor-based (e.g., silicon-based) devices. In this example, the programmable circuit system 1412 implements an example model observation circuit system 492, an example observation limiting circuit system 494, an example model parameter quantization circuit system 496, an example device configuration platform 425, and more generally, an example model quantizer circuit system 405.

[0154] The illustrated programmable circuit system 1412 includes local memory 1413 (e.g., cache, registers, etc.). The illustrated programmable circuit system 1412 communicates via bus 1418 with main memory 1414, 1416, which includes volatile memory 1414 and non-volatile memory 1416. The volatile memory 1414 may be one or more synchronous dynamic random access memories (SDRAM) or dynamic random access memories (DRAM). Dynamic Random Access Memory Or any other type of RAM device. The non-volatile memory 1416 may be implemented by flash memory or any other desired type of memory device, or a combination thereof. Access to the main memory 1414, 1416 of the illustrated example is controlled by the memory controller 1417. In some examples, the memory controller 1417 may be implemented by one or more integrated circuits, logic circuits, microcontrollers, or any other type of circuit system from any desired series or manufacturer to manage the flow of data to and from the main memory 1414, 1416.

[0155] The illustrated programmable circuit system platform 1400 also includes an interface circuit system 1420. The interface circuit system 1420 can be implemented in hardware according to any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB) interface, etc. Interfaces include Near Field Communication (NFC) interfaces, Peripheral Component Interconnect (PCI) interfaces, or Peripheral Component Interconnect High Speed ​​(PCIe) interfaces.

[0156] In the illustrated example, one or more input devices 1422 are connected to the interface circuitry 1420. The input devices 1422 allow a user (e.g., a human user, a machine user, etc.) to input one or a combination of data or commands into the programmable circuitry 1412. The input devices 1422 may be implemented as one or a combination of, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touchscreen, a trackpad, a trackball, a dot device, or a voice recognition system.

[0157] One or more output devices 1424 are also connected to the interface circuitry 1420 of the illustrated example. The output devices 1424 may be implemented, for example, by one or a combination of a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-situ switching (IPS) display, a touchscreen, etc.), a haptic output device, a printer, or a speaker. Therefore, the interface circuitry 1420 of the illustrated example includes one or a combination of a graphics driver card, a graphics driver chip, or a graphics processor circuitry system such as a GPU.

[0158] The illustrated interface circuit system 1420 also includes communication devices, such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point, or network interface, or a combination thereof, to facilitate the exchange of data with external machines (e.g., any kind of computing device) via network 1426. Communication can be made via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-of-sight wireless systems, line-of-sight wireless systems, cellular telephone systems, optical connections, etc.

[0159] The illustrated programmable circuit system platform 1400 also includes one or more mass storage disks or devices 1428 for storing one or more of firmware, software, or data. Examples of such mass storage disks or devices 1428 include one or more magnetic storage devices (e.g., floppy disks, drives, HDDs, etc.), optical storage devices (e.g., Blu-ray discs, CDs, DVDs, etc.), RAID systems, or solid-state storage disks or devices (e.g., flash memory devices and SSDs).

[0160] can be Figures 10 to 11 The machine-readable instruction 1432 implemented by the machine-readable instruction can be stored in one or a combination of the following: in a mass storage device 1428, in a volatile memory 1414, in a non-volatile memory 1416, or on at least one non-transitory computer-readable storage medium that may be removable, such as a CD or DVD.

[0161] Figure 15 yes Figure 14 A block diagram of an example embodiment of the programmable circuit system 1412. In this example, Figure 14 The programmable circuit system 1412 is implemented by the microprocessor 1500. For example, the microprocessor 1500 may be a general-purpose microprocessor (e.g., a general-purpose microprocessor circuit system). The microprocessor 1500 executes... Figures 10 to 11 The flowchart contains some or all of the machine-readable instructions to effectively translate... Figure 2 The circuit system is instantiated as a logic circuit to perform operations corresponding to those machine-readable instructions. In some such examples, Figure 4 The circuit system is instantiated by the hardware circuitry of microprocessor 1500 in conjunction with machine-readable instructions. For example, microprocessor 1500 may be implemented by a multi-core hardware circuitry system such as a CPU, DSP, GPU, or XPU. While the microprocessor may contain any number of example cores 1502 (e.g., one core), this example microprocessor 1500 is a multi-core semiconductor device containing N cores. The cores 1502 of microprocessor 1500 may operate independently or collaboratively to execute machine-readable instructions. For example, machine code corresponding to firmware, embedded software, or software programs may be executed by one of the cores 1502, or by multiple cores 1502 at the same or different times. In some examples, the machine code corresponding to firmware, embedded software, or software programs is divided into threads and executed in parallel by two or more of the cores 1502. Software programs may correspond to... Figures 10 to 11 A flowchart represents part or all of machine-readable instructions or operations.

[0162] Core 1502 can communicate via a first example bus 1504. In some examples, the first bus 1504 can be implemented as a communication bus to enable communication associated with the device in core 1502. For example, the first bus 1504 can be implemented via at least one of an Interconnect Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Alternatively, the first bus 1504 can be implemented by any other type of computing or electrical bus. Core 1502 can obtain data, instructions, and signals from one or more external devices via example interface circuitry 1506. Core 1502 can output data, instructions, and signals to one or more external devices via interface circuitry 1506. While the core 1502 of this example includes example local memory 1520 (e.g., a Level 1 (L1) cache, which may be divided into an L1 data cache and an L1 instruction cache), the microprocessor 1500 also includes example shared memory 1510 (e.g., a Level 2 (L2) cache) that can be shared by the cores for high-speed access to data and instructions. Data and instructions can be transferred (e.g., shared) by writing to or reading from the shared memory 1510. The local memory 1520 and shared memory 1510 of each of the cores 1502 may be multi-level cache memory and main memory (e.g., Figure 14 The cache is part of the storage device hierarchy of main memories (1414, 1416). Typically, higher-level memories in the hierarchy exhibit shorter access times and smaller storage capacities compared to lower-level memories. Changes at each level of the cache hierarchy are managed by cache coherence strategies (e.g., coordination).

[0163] Each core 1502 may be referred to as a CPU, DSP, GPU, or any other type of hardware circuitry. Each core 1502 includes a control unit circuitry 1514, an arithmetic and logic (AL) circuitry (sometimes called an ALU) 1516, multiple registers 1518, local memory 1520, and a second example bus 1522. Other structures may exist. For example, each core 1502 may include a vector unit circuitry, a single instruction multiple data (SIMD) unit circuitry, a load / store unit (LSU) circuitry, a branch / jump unit circuitry, a floating-point unit (FPU) circuitry, etc. The control unit circuitry 1514 includes semiconductor-based circuitry constructed to control (e.g., coordinate) the movement of data within the corresponding core 1502. The AL circuitry 1516 includes semiconductor-based circuitry constructed to perform one or more mathematical or logical operations on the data within the corresponding core 1502. Some examples of the AL circuitry 1516 perform integer-based operations. In other examples, the AL circuitry 1516 also performs floating-point operations. In other examples, the AL circuit system 1516 may include a first AL circuit system that performs integer-based operations and a second AL circuit system that performs floating-point operations. In some examples, the AL circuit system 1516 may be referred to as an arithmetic logic unit (ALU).

[0164] Register 1518 is a semiconductor-based structure used to store data and instructions, such as the results of one or more operations performed by the AL circuitry 1516 corresponding to core 1502. For example, register 1518 may include vector registers, SIMD registers, general-purpose registers, flag registers, segment registers, machine-specific registers, instruction pointer registers, control registers, debug registers, memory management registers, machine check registers, etc. Register 1518 may be located in memory, such as... Figure 15 As shown. Alternatively, register 1518 can be organized in any other arrangement, format, or structure, for example, by distributing it within core 1502 to reduce access time. The second bus 1522 can be implemented by at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0165] Each core 1502, or more generally, the microprocessor 1500 may include additional or alternative structures to those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more convergent / common grid stoppers (CMS), one or more shifters (e.g., barrel shifters), or other circuitry may be present. The microprocessor 1500 is a semiconductor device manufactured to include a plurality of transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) contained in one or more packages.

[0166] The microprocessor 1500 may include or cooperate with one or more accelerators (e.g., acceleration circuitry, hardware accelerators, etc.). In some examples, accelerators are implemented by logic circuitry to perform certain tasks faster and more efficiently than a general-purpose processor can. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. GPUs, DSPs, or other programmable devices may also serve as accelerators. Accelerators may be mounted on the microprocessor 1500, in the same chip package as the microprocessor 1500, or in one or more separate packages with the microprocessor 1500.

[0167] Figure 16 yes Figure 14 A block diagram of another example embodiment of the programmable circuit system 1412 is shown. In this example, the programmable circuit system 1412 is implemented by an FPGA circuit system 1600. For example, the FPGA circuit system 1600 may be implemented by an FPGA. The FPGA circuit system 1600 can be used, for example, to execute machine-readable instructions that can be executed by a corresponding FPGA. Figure 15 The example microprocessor 1500 performs operations in other ways. However, once configured, the FPGA circuit system 1600 instantiates operations and functions corresponding to machine-readable instructions in hardware, and therefore can generally perform operations / functions faster than those that can be performed by a general-purpose microprocessor executing the corresponding software.

[0168] More specifically, as described above Figure 15 The microprocessor 1500 (which is programmable to execute by) Figures 10 to 11 The flowchart represents some or all of the general-purpose devices in machine-readable instructions, but whose interconnection and logic circuitry system are fixed once manufactured. Figure 16 The example FPGA circuit system 1600 includes interconnect and logic circuit systems that can be configured, constructed, programmed, and interconnected in different ways after manufacturing to instantiate, for example, a system corresponding to a system derived from a FPGA. Figures 10 to 11The flowchart represents some or all of the operations / functions of machine-readable instructions. Specifically, the FPGA circuit system 1600 can be considered as an array of logic gates, interconnects, and switches. Switches can be programmed to change how logic gates are interconnected via interconnects, thereby effectively forming one or more dedicated logic circuits (unless and until the FPGA circuit system 1600 is reprogrammed). The logic circuits are configured so that logic gates can cooperate in different ways to perform different operations on data received from the input circuit system. Those operations can correspond to... Figures 10 to 11 The flowchart represents some or all of the instructions (e.g., software and / or firmware). Therefore, the FPGA circuit system 1600 can be configured or constructed to effectively transmit the corresponding... Figures 10 to 11 Some or all of the machine-readable instructions in the flowchart are instantiated into at least one of the special-purpose logic circuits to perform the operations / functions corresponding to those software instructions in a special-purpose manner similar to that of an ASIC. Therefore, the FPGA circuit system 1600 can be coupled with a general-purpose microprocessor capable of executing operations / functions corresponding to those software instructions. Figures 10 to 11 Some or all of the operations / functions in the machine-readable instructions can be performed faster than the operations / functions in the machine-readable instructions.

[0169] exist Figure 16 In some examples, the FPGA circuit system 1600 is configured or constructed in response to being programmed (and / or reprogrammed one or more times) based on a binary file. In some examples, the binary file may be compiled or generated based on instructions in, for example, Lucid's Hardware Description Language (HDL), the Very High Speed ​​Integrated Circuit (VHSIC) Hardware Description Language (VHDL), or Verilog. For example, a user (e.g., a human user, a machine user, etc.) may write code or programs corresponding to one or more operations / functions in the HDL; the code / program may be translated into a low-level language as needed; and the code / program (e.g., code / program in a low-level language) may be translated into a binary file (e.g., by a compiler, software application, etc.). In some examples, Figure 16 The FPGA circuit system 1600 can access or load at least one of the binary files to enable Figure 16 The FPGA circuit system 1600 is configured or constructed to perform at least one of one or more operations / functions. For example, a binary file can be generated by... Figure 16 The FPGA circuit system 1600 can access one or a combination of bit streams (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.) or machine-readable instructions for configuration or construction. Figure 16 At least one of the FPGA circuit system 1600 or a portion thereof.

[0170] In some examples, the binary file is at least one of those compiled, generated, transformed, or otherwise output from a unified software platform used for programming the FPGA. For example, the unified software platform can transform first instructions (e.g., code or program) corresponding to one or more operations / functions in a high-level language (e.g., C, C++, Python, etc.) into second instructions corresponding to one or more operations / functions in an HDL. In some such examples, the binary file is at least one of those compiled, generated, or otherwise output from the unified software platform based on the second instructions. In some examples, Figure 16 The FPGA circuit system 1600 can access or load at least one of the binary files to enable Figure 16 The FPGA circuit system 1600 is configured or constructed to perform at least one of one or more operations / functions. For example, a binary file can be generated by... Figure 16 The FPGA circuit system 1600 can access one or a combination of bit streams (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), data (e.g., computer-readable data, machine-readable data, etc.) or machine-readable instructions for configuration or construction. Figure 16 At least one of the FPGA circuit system 1600 or a portion thereof.

[0171] Figure 16 The FPGA circuit system 1600 includes an example input / output (I / O) circuit system 1602 to obtain data from or output data to at least one of the example configuration circuit system 1604 or external hardware 1606. For example, the configuration circuit system 1604 may be implemented by an interface circuit system that provides a binary file, which may be implemented by one or more of bitstreams, data, or machine-readable instructions to configure the FPGA circuit system 1600 or a portion thereof. In some such examples, the configuration circuit system 1604 may obtain the binary file from a user, a machine (e.g., a hardware circuit system (e.g., a programmable or dedicated circuit system) that can implement an artificial intelligence / machine learning (AI / ML) model to generate a binary file), or any combination thereof. In some examples, the external hardware 1606 may be implemented by an external hardware circuit system. For example, the external hardware 1606 may be implemented by... Figure 15 The microprocessor 1500 is implemented.

[0172] The FPGA circuit system 1600 also includes an array of example logic gate systems 1608, multiple example configurable interconnects 1610, and example memory circuit systems 1612. The logic gate systems 1608 and configurable interconnects 1610 can be configured to instantiate corresponding to... Figures 10 to 11One or more operations / functions in at least some of the machine-readable instructions, and / or other desired operations. Figure 16 The logic gate system 1608 shown is manufactured in blocks or groups. Each block contains semiconductor-based electrical structures that can be configured into logic circuits. In some examples, the electrical structures contain logic gates (e.g., AND gates, OR gates, NOR gates, etc.) that provide basic building blocks for the logic circuits. Electrically controlled switches (e.g., transistors) are present in each of the logic gate system 1608 to enable the configuration of one or a combination of electrical structures or logic gates to form a circuit for performing a desired operation / function. The logic gate system 1608 may contain other electrical structures such as lookup tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc.

[0173] The configurable interconnect 1610 of the illustrated example is a conductive path, trace, via, etc., that may contain electrically controlled switches (e.g., transistors), the state of which can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuit system 1608 to program the desired logic circuit.

[0174] The illustrated storage circuit system 1612 is structured to store the results of one or more operations performed by corresponding logic gates. The storage circuit system 1612 may be implemented using registers, etc. In the illustrated example, the storage circuit system 1612 is distributed within the logic gate circuit system 1608 to facilitate access and improve execution speed.

[0175] Figure 16 The example FPGA circuit system 1600 also includes an example dedicated operating circuit system 1614. In this example, the dedicated operating circuit system 1614 includes a dedicated circuit system 1616, which can be invoked to implement common functions to avoid the need for field programming of those functions. Examples of such dedicated circuit systems 1616 include memory (e.g., DRAM) controller circuit systems, PCIe controller circuit systems, clock circuit systems, transceiver circuit systems, memory and multiplier-accumulator circuit systems. Other types of dedicated circuit systems may be present. In some examples, the FPGA circuit system 1600 may also include an example general-purpose programmable circuit system 1618, such as an example CPU 1620 or an example DSP 1622. Other general-purpose programmable circuit systems 1618, such as GPUs, XPUs, etc., that can be programmed to perform other operations may also be present, either additionally or alternatively.

[0176] although Figure 15 and 16 illustrate Figure 14Two example implementations of the programmable circuit system 1412 are provided, but many other approaches are envisioned. For example, the FPGA circuit system may include an onboard CPU, such as... Figure 15 One or more of the example CPUs 1620. Therefore, Figure 14 The programmable circuit system 1412 can also be configured by at least combining Figure 15 Example microprocessor 1500 and Figure 16 The example FPGA circuit system 1600 is used for implementation. In some such hybrid examples, Figure 15 One or more cores of 1502 executable by Figures 10 to 11 The flowchart represents the first part of machine-readable instructions to perform a first operation / function. Figure 16 The FPGA circuit system 1600 can be configured or constructed to perform operations corresponding to those performed by... Figures 10 to 11 The flowchart represents at least one of the second operations / functions in the second part of the machine-readable instructions, and / or the ASIC can be configured or constructed to perform operations corresponding to those performed by the second part of the machine-readable instructions. Figures 10 to 11 The flowchart represents at least one of the third operations / functions in the third part of a machine-readable instruction.

[0177] therefore, Figure 4 Some or all of the circuitry in a system can be instantiated at the same or different times. For example, Figure 15 The same and / or different parts of the microprocessor 1500 can be programmed to execute machine-readable instructions at the same and / or different times. In some examples, Figure 16 The same and / or different parts of the FPGA circuit system 1600 can be configured or constructed to perform at least one of the operations / functions corresponding to the machine-readable instructions at the same and / or different times.

[0178] In some examples, Figure 4 Some or all of the circuit system can be instantiated, for example, in one or more threads that execute in parallel and / or serially. Figure 15 The microprocessor 1500 can execute machine-readable instructions in one or more threads that execute in parallel and / or serially. In some examples, Figure 16 The FPGA circuit system 1600 can be configured or constructed to perform at least one of operations / functions in parallel and / or serially. Furthermore, in some examples, Figure 4 Some or all of the circuit systems can be in Figure 15 It is implemented within one or more virtual machines or containers and executed on a microprocessor 1500.

[0179] In some examples, Figure 14 The programmable circuit system 1412 can be housed in one or more packages. For example, Figure 15 microprocessor 1500 or Figure 16 At least one of the FPGA circuitry systems 1600 may be housed in one or more packages. In some examples, the XPU may be derived from... Figure 14 The programmable circuit system 1412 is implemented, and the programmable circuit system may be in one or more packages. For example, the XPU may be contained within a CPU in one package (e.g., Figure 15 Microprocessor 1500, Figure 16 CPU 1620, etc.), and DSP in another package (e.g., Figure 16 DSP 1622), GPU in another package, and FPGA in yet another package (e.g., Figure 16 FPGA circuit system 1600).

[0180] Figure 17 The flowchart in the middle describes the software (e.g., Figure 14 Example software distribution platform 1705 distributes example machine-readable instructions (1432) to other hardware devices (e.g., one or more hardware devices owned or operated by a third party from the owner or operator of the software distribution platform). Example software distribution platform 1705 may be implemented by any computer server, data facility, cloud service, etc., capable of storing software and transferring software to other computing devices. The third party may be a customer of an entity that owns or operates at least one of the software distribution platforms 1705. For example, an entity that owns or operates at least one of the software distribution platforms 1705 may be a software (e.g., Figure 14 The machine-readable instruction 1432 is provided by at least one of the developers, sellers, or licensors. A third party may be a consumer, user, retailer, OEM, etc., who purchases or licenses one or a combination of the software for at least one of use, resale, or sublicensing. In the illustrated example, the software distribution platform 1705 includes one or more servers and one or more storage devices. The storage devices store the machine-readable instruction 1432, which may correspond to... Figures 10 to 11 Example machine-readable instructions, as described above. One or more servers of the example software distribution platform 1705 communicate with the example network 1710, which may correspond to any one or more of the Internet or any of the example networks described above. In some examples, one or more servers respond to a request to transfer software to a requesting party as part of a commercial transaction. Payment for at least one of the software delivery, sale, or licensing may be handled by one or more servers of at least one of the software distribution platforms or by a third-party payment entity. The servers enable one or more purchasers or licensors to download machine-readable instructions 1432 from the software distribution platform 1705. For example, it may correspond to... Figures 10 to 11Software containing example machine-readable instructions can be downloaded to example programmable circuit system platform 1400, which will execute machine-readable instructions 1432 to implement model quantizer circuit system 405. In some examples, one or more servers of software distribution platform 1705 periodically distribute the software (e.g., Figure 14 Example machine-readable instruction 1432) provides, transmits, or causes at least one of the following to ensure that improvements, patches, updates, etc., are distributed and applied to the software at the end-user device. Although referred to as software above, distributed “software” may alternatively be firmware.

[0181] Although Figure 4 The example shown illustrates the implementation of the model quantizer circuit system 405 in Figure 1, but... Figure 4 One or more of the elements, processes, or devices described herein may be combined, divided, rearranged, omitted, removed, or implemented in any other way. Furthermore, the example model observation circuit system 492, the example observation limiting circuit system 494, the example model parameter quantization circuit system 496, the example device configuration platform 425, or more generally, Figure 4 The example model quantizer circuit system 405 can be implemented by hardware alone or by a combination of hardware, software, and firmware. Therefore, for example, any of the example model observation circuit system 492, example observation limiting circuit system 494, example model parameter quantization circuit system 496, example device configuration platform 425, or more generally, any of the example model quantizer circuit systems 405 can be implemented by a programmable circuit system in combination with one or more machine-readable instructions (e.g., firmware or software), processor circuit system, analog circuitry, digital circuitry, logic circuitry, programmable processor, programmable microcontroller, graphics processing unit (GPU), digital signal processor (DSP), ASIC, programmable logic device (PLD), or field-programmable logic device (FPLD) (e.g., FPGA). Furthermore, Figure 4 Example model quantizer circuit system 405 may include, in addition to or instead of Figure 4 It may include one or more of the elements, processes or devices described herein, or may include more than one of any or all of the elements, processes and devices described herein.

[0182] Figures 10 to 11 The diagram illustrates example machine-readable instructions or flowcharts representing example operations, which can be executed by a programmable circuit system for implementation or instantiation. Figure 4 At least one of the model quantizer circuit systems 405, the example operation can be executed by a programmable circuit system for implementation or instantiation. Figure 4At least one of the model quantizer circuit systems 405. Machine-readable instructions may be provided by a programmable circuit system (e.g., in conjunction with the following). Figure 14 The programmable circuit system 1412 shown in the example processor platform 1400 discussed here executes one or more executable programs or portions of one or more executable programs, and may be used by the following combined Figure 15 Or, as discussed in example 16, a programmable circuit system (e.g., an FPGA) performs one or more functions or portions of functions. In some examples, machine-readable instructions cause operations, tasks, etc., to be performed or executed in a real-world manner. As used herein, “automation” means without human intervention.

[0183] The program may be embodied in instructions (e.g., software and / or firmware) stored on one or more non-transitory computer-readable and / or machine-readable storage media, such as cache memory, magnetic storage devices or disks (e.g., floppy disks, hard disk drives (HDDs), etc.), optical storage devices or optical discs (e.g., Blu-ray discs, optical discs (CDs), digital versatile discs (DVDs), etc.), redundant arrays of independent disks (RAID), registers, ROM, solid-state drives (SSDs), SSD memory, non-volatile memory (e.g., electrically erasable programmable read-only memory (EEPROM), flash memory, etc.), volatile memory (e.g., any type of random access memory (RAM), etc.), or any other storage device or storage disk, or a combination thereof. The instructions of the non-transitory computer-readable and / or machine-readable media may be programmed or executed by a programmable circuit system located in one or more hardware devices, but the entire program or a portion thereof may alternatively be executed or instantiated or embodied in one or more hardware devices other than the programmable circuit system. Machine-readable instructions may be distributed across multiple hardware devices or executed by two or more hardware devices (e.g., server and client hardware devices). For example, client hardware devices may be implemented by endpoint client hardware devices (e.g., hardware devices associated with human and / or machine users) or by an intermediate client hardware device gateway (e.g., a radio access network (RAN)) that facilitates communication between the server and endpoint client hardware devices. Similarly, non-transitory computer-readable storage media may contain one or more media. Furthermore, although references... Figures 10 to 11The flowcharts described herein are example programs, but many other methods can be used to implement the example model quantizer circuit system 405. For example, the execution order of blocks in the flowchart can be changed, or some of the described blocks can be altered, eliminated, or combined. Furthermore or alternatively, any or all of the blocks in the flowchart can be implemented by one or more hardware circuits (e.g., processor circuit systems, discrete, integrated analog and / or digital circuit systems, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. The programmable circuit systems can be distributed across different network locations or local to one or more hardware devices (e.g., single-core processors (e.g., single-core CPUs), multi-core processors (e.g., multi-core CPUs, XPUs, etc.)). For example, a programmable circuit system may be one or a combination of the following: a CPU or FPGA located in the same package (e.g., in the same integrated circuit (IC) package or in two or more separate housings), one or more processors in a single machine, multiple processors distributed across multiple servers across server racks, multiple processors distributed across one or more server racks, or any combination thereof.

[0184] The machine-readable instructions described herein can be stored in one or more of the following formats: compressed format, encrypted format, segmented format, compiled format, executable format, and encapsulated format. The machine-readable instructions described herein can be stored as data (e.g., computer-readable data, machine-readable data, one or more bits (e.g., one or more computer-readable bits, one or more machine-readable bits, etc.), bit streams (e.g., computer-readable bit streams, machine-readable bit streams, etc.)) or data structures (e.g., as parts of instructions, code, code representations, etc.), which can be used to create, manufacture, or generate machine-executable instructions. For example, machine-readable instructions can be segmented and stored on one or more storage devices, disks, or computing devices (e.g., servers) located at the same or different locations within a network or network set (e.g., in the cloud, edge devices, etc.). Machine-readable instructions may require one or more of the following to be installed, modified, adapted, updated, combined, supplemented, configured, decrypted, decompressed, decapsulated, allocated, reallocated, compiled, etc., so that they are directly readable, interpretable, or executable by computing devices and / or other machines. For example, machine-readable instructions may be stored in multiple parts, which are individually compressed, encrypted, or stored on separate computing devices, wherein the parts, when decrypted, decompressed, or combined, form a set of one or more computer-executable or machine-executable instructions, the implementation of which together form one or more functions or operations of a program, such as those described herein.

[0185] In another example, machine-readable instructions may be stored in a state that is readable by a programmable circuit system, but require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., to execute the machine-readable instructions on a particular computing device or another device. In another example, the machine-readable instructions or corresponding programs may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before they can be executed fully or partially. Therefore, as used herein, machine-readable, computer-readable, or machine-readable media may contain one or a combination of instructions or programs, regardless of the specific format or state of the machine-readable instructions or programs.

[0186] The machine-readable instructions described in this article can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0187] As mentioned above, Figures 10 to 11Example operations can be implemented using executable instructions (e.g., computer-readable and / or machine-readable instructions) stored on one or more non-transitory computer-readable or machine-readable media. As used herein, the terms non-transitory computer-readable media, non-transitory computer-readable storage media, non-transitory machine-readable media, and non-transitory machine-readable storage media are explicitly defined as containing any type of computer-readable storage device or disk and not containing propagating signals or transmission media. Examples of such non-transitory computer-readable media, non-transitory computer-readable storage media, non-transitory machine-readable media, or non-transitory machine-readable storage media include one or more optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, or any other storage device or disk where information is stored for any duration (e.g., a prolonged period of time, permanently, temporarily, temporarily buffered, cached information). As used herein, the terms "non-transitory computer-readable storage device" and "non-transitory machine-readable storage device" are defined as comprising any physical (mechanical, magnetic, electromechanical, or electrical) hardware for retaining information for a period of time, but excluding the propagation of signals and the transmission medium. Examples of non-transitory computer-readable storage devices or non-transitory machine-readable storage devices include one or a combination of the following: any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disk, magnetic disk, disk drive, or redundant array of independent disks (RAID) system. As used herein, the term "device" refers to a physical structure, such as one or a combination of the following: mechanical, electromechanical, or electrical equipment, hardware, or circuitry that may or may not be configured by, or manufactured to execute, computer-readable instructions, machine-readable instructions, etc.

[0188] "Including" and "comprising" (and all their forms and tenses) are used as open terms herein. Therefore, whenever a technical solution uses any form of "including" or "comprising" (e.g., includes, includes, comprising, having, etc.) as a preposition or in any kind of technical solution citation, additional elements, terms, etc., may exist without exceeding the scope of the corresponding technical solution or citation. As used herein, when the phrase "at least" is used as a transitional term in, for example, a technical solution preposition, it is open in the same way as the terms "including" and "comprising" are open. The term "and / or," when used in the form of, for example, A, B, and / or C, refers to any combination or subset of A, B, C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, projects, objects, and things, the phrase “at least one of A and B” means an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, projects, objects, and things, the phrase “at least one of A or B” means an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or implementation of processes, instructions, actions, activities, etc., the phrase “at least one of A and B” means an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the execution or implementation of processes, instructions, actions, activities, etc., the phrase “at least one of A or B” means an implementation that includes any one of the following: (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0189] As used herein, singular references (e.g., "a(a)", "an(an)", "first", "second", etc.) do not exclude plurals. As used herein, the term "a" or "an" refers to one or more of the objects. The terms "a" (or "an"), "one or more", and "at least one" are used interchangeably herein. Furthermore, although listed separately, multiple means, elements, or actions may be implemented by, for example, the same entity or object. Moreover, while individual features may be included in different examples or solutions, these features may be combined, and inclusion in different examples or solutions does not imply that the combination of features is infeasible or disadvantageous at least one of them.

[0190] As used herein, unless otherwise indicated, a connection reference (e.g., attachment, coupling, connection, and joining) may include an intermediate member between elements referenced by at least one of the connection references or relative movements between those elements. Therefore, a connection reference does not necessarily imply that two elements are directly connected or fixed to each other. As used herein, stating that any part is "in contact" with another part is defined as meaning that there is no intermediate part between the two parts.

[0191] Unless otherwise specifically stated, descriptive terms such as “first,” “second,” and “third” are used herein without intending to or otherwise indicate priority, physical order, arrangement, or any sorting in the list, but are used only as markers or arbitrary names to distinguish elements in order to facilitate understanding of the described examples. In some examples, the descriptive term “first” may be used to refer to an element in a particular embodiment, while the same element may be referred to in the technical solution by different descriptive terms such as “second” or “third.” In such cases, such descriptive terms are used only to clearly identify those elements within the context of the discussion (e.g., within the technical solution), where elements may otherwise share the same name.

[0192] As used herein, “approximately” and “about” modify their subject / value to identify the potential for variation that may occur in real-world applications. For example, “approximately” and “about” may modify dimensions that may be imprecise due to at least one of manufacturing tolerances or other real-world defects. For example, unless otherwise specified herein, “approximately” and “about” may indicate that such dimensions are within a tolerance of + / - 10%.

[0193] As used in this article, "substantially real-time" means occurring in a near-instantaneous manner, taking into account real-world delays such as computation time and transmission. Therefore, unless otherwise specified, "substantially real-time" means real-time plus 1 second.

[0194] As used herein, the phrase “communication” includes variations thereof, encompassing one or a combination of direct communication or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication or constant communication, but also includes selective communication at at least one of periodic intervals, predetermined intervals, non-periodic intervals or one-off events.

[0195] As used herein, a “programmable circuit system” is defined as comprising at least one of the following: (i) one or more special-purpose circuits (e.g., special-purpose circuits (ASICs)) configured to perform a particular operation and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), or (ii) one or more general-purpose semiconductor-based circuits programmable by instructions to perform one or more particular functions or operations and comprising one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of programmable circuit systems include programmable microprocessors, such as: a central processing unit (CPU) that can execute first instructions to perform one or more operations or functions; a field-programmable gate array (FPGA) that can be programmed with second instructions to configure or structure at least one of the FPGAs, thereby instantiating one or more operations or functions corresponding to the first instructions; a graphics processing unit (GPU) that can execute first instructions to perform one or more operations or functions; a digital signal processor (DSP) that can execute first instructions to perform one or more operations or functions; an XPU; a network processing unit (NPU); one or more microcontrollers that can execute first instructions to perform one or more operations or functions; or an integrated circuit, such as an application-specific integrated circuit (ASIC). For example, an XPU can be implemented by a heterogeneous computing system that includes a variety of programmable circuit systems (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more NPUs, one or more DSPs, etc., and any combination thereof) and configuration technologies (e.g., application programming interfaces (APIs)) that can assign computing tasks to any type of programmable circuit system that is suitable and can be used to perform the computing tasks.

[0196] As used herein, an integrated circuit / circuit system is defined as one or more semiconductor packages containing one or more circuit elements, such as transistors, capacitors, inductors, resistors, current paths, diodes, etc. For example, an integrated circuit can be implemented as one or more of an ASIC, FPGA, chip, microchip, programmable circuit system, semiconductor substrate coupling multiple circuit elements, system-on-a-chip (SoC), etc.

[0197] In this specification, the term "coupled" may encompass a connection, communication, or signal path that achieves a functional relationship consistent with this specification. For example, if device A generates a signal to control device B to perform an action, then: (a) in a first example, device A is coupled to device B via a direct connection; or (b) in a second example, device A is coupled to device B via an intermediate component C, provided that the intermediate component C does not alter the functional relationship between device A and device B, such that device B is controlled by device A via the control signal generated by device A.

[0198] A device “configured to” perform a task or function may be configured (e.g., at least one of programming or hardwiring) to perform the function during manufacturing by the manufacturer, or may be configured (or reconfigurable) by the user after manufacturing to perform the function or at least one of other additional or alternative functions. The configuration may be performed by at least one of firmware or software programming of the device, by at least one of the construction or layout of the device’s hardware components and interconnects, or by a combination thereof.

[0199] As used herein, the terms “terminal,” “node,” “interconnect,” “pin,” and “lead” are used interchangeably. Unless specifically stated otherwise, these terms are generally used to refer to interconnections or their terminations between device elements, circuit elements, integrated circuits, devices, or other electronic or semiconductor components.

[0200] In the description and technical solutions, the described "circuit system" may include one or more circuits. A circuit or device described herein as containing certain components may be practically adapted to be coupled to those components to form the described circuit system or device. For example, a structure described as containing one or more semiconductor elements (e.g., transistors), one or more passive elements (e.g., one or a combination of resistors, capacitors, or inductors), or one or more sources (e.g., voltage sources and / or current sources) may alternatively contain only semiconductor elements within a single physical device (e.g., at least one of a semiconductor die or integrated circuit (IC) package) and may be adapted to be coupled to at least some of the passive elements or sources during or after manufacturing, for example, by at least one of an end user or a third party, to form the described structure.

[0201] The circuits described herein can be reconfigured to include replacement components to provide functionality at least partially similar to that available before the component replacement. Unless otherwise stated, components shown as resistors generally represent one or more elements coupled in at least one of series or parallel to provide the amount of impedance represented by the resistor shown. For example, a resistor or capacitor shown and described herein as a single component may alternatively be multiple resistors or capacitors coupled in parallel between the same nodes. For example, a resistor or capacitor shown and described herein as a single component may actually be multiple resistors or capacitors coupled in series between the same two nodes as the single resistor or capacitor. While some elements in the described examples are included in the integrated circuit and others are outside the integrated circuit, in other example embodiments, additional or fewer features may be incorporated into the integrated circuit. Additionally, some or all features described as being outside the integrated circuit may be included in the integrated circuit, and some features described as being inside the integrated circuit may be incorporated outside the integrated circuit. As used herein, the term "integrated circuit" means one or more circuits that are at least one of the following: (i) incorporated in / above a semiconductor substrate; (ii) incorporated in a single semiconductor package; (iii) incorporated in the same module; or (iv) incorporated in / on the same printed circuit board.

[0202] The use of the phrase “grounding” in the foregoing description includes at least one of chassis grounding, ground wire grounding, floating grounding, virtual grounding, digital grounding, general grounding, or any other form of grounding connection applicable to or suited to the teachings of this specification. Unless otherwise stated, “about,” “approximately,” or “substantially” preceding a value indicates + / - 10% of said value, or, if the value is zero, a reasonable range of values ​​near zero.

[0203] Within the scope of the technical solution, modifications can be made to the described embodiments, and other embodiments are possible.

[0204] As can be understood from the foregoing, example systems, apparatuses, artifacts, and methods for implementing outlier removal to quantize machine learning models (e.g., transformer networks) have been described. The described systems, apparatuses, artifacts, and methods improve the efficiency of machine learning models implemented by a target device by removing outliers from the weights and activations of the floating-point machine learning model observed during quantization. By removing such outliers, the range of values ​​to be represented on the target device by the weights and activations of the fixed-point machine learning model is reduced. This can lead to improved model error and / or model accuracy compared to other model quantization techniques. The described systems, apparatuses, artifacts, and methods also relate to one or more improvements in the operation of machines (e.g., computers or other electronic devices implementing machine learning models).

[0205] Other examples and combinations thereof include the following. Example 1 includes a non-transitory computer-readable medium comprising computer-readable instructions to cause at least one processor circuitry to at least: limit the values ​​of activations associated with layers of a floating-point version of a machine learning model to determine a limited value of the activations, the values ​​of the activations being based on calibration data applied to the floating-point version of the machine learning model; determine a quantization factor using the limited value of the activations to quantize activations associated with corresponding layers of a fixed-point version of the machine learning model; and configure the fixed-point version of the machine learning model on a device using the quantization factor.

[0206] Example 2 includes a non-transitory computer-readable medium according to Example 1, wherein the instructions cause one or more of the at least one processor circuitry to initiate the execution of the floating-point version of the machine learning model using the calibration data, and cause the activated, throttled value to propagate to subsequent layers of the floating-point version of the machine learning model during the execution.

[0207] Example 3 includes a non-transitory computer-readable medium according to Example 1, wherein the activation is a first activation, and the instructions will cause one or more of the at least one processor circuitry to: observe values ​​of a plurality of activations associated with the layer of the floating-point version of the machine learning model, the values ​​of the plurality of activations being based on calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; determine a metric using the values ​​of the plurality of activations; and limit the value of the first activation using the metric.

[0208] Example 4 includes a non-transitory computer-readable medium according to Example 3, wherein the instructions cause one or more of the at least one processor circuitry to scale the metric to determine a scaled metric, and use the scaled metric to limit the value of the first activation.

[0209] Example 5 includes a non-transitory computer-readable medium according to Example 3, wherein the metric is the standard deviation of the values ​​of the plurality of activations.

[0210] Example 6 includes a non-transitory computer-readable medium according to Example 5, wherein the instructions cause one or more of the at least one processor circuitry to determine the mean of the plurality of activated values, and to limit the value of the first activated value by multiplying the mean and the standard deviation by a number.

[0211] Example 7 includes a non-transitory computer-readable medium according to Example 3, wherein the plurality of activations correspond to a single channel associated with the layer of the floating-point version of the machine learning model.

[0212] Example 8 includes a non-transitory computer-readable medium according to Example 1, wherein the activation is a first activation, the quantization factor includes a scaling factor, and the instructions will cause one or more of the at least one processor circuitry to determine the scaling factor by: determining a range of observations for a plurality of activations associated with the layer of the floating-point version of the machine learning model, the observations for the plurality of activations being based on calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; and determining the scaling factor using a ratio of the range of observations to the quantization range associated with the corresponding layer of the fixed-point version of the machine learning model.

[0213] Example 9 includes a non-transitory computer-readable medium according to Example 8, wherein the quantization factor includes an offset factor, and the instructions cause one or more of the at least one processor circuitry to determine the offset factor using a ratio of a first observation among the observations to the scaling factor.

[0214] Example 10 includes a non-transitory computer-readable medium according to Example 1, wherein the quantization factor is a first quantization factor, and the instructions will cause one or more of the at least one processor circuitry to: observe the values ​​of a first plurality of weights associated with the layer of the floating-point version of the machine learning model, the first plurality of weights corresponding to a single channel associated with the layer of the floating-point version of the machine learning model; limit the value of the first weight among the first plurality of weights using a metric based on the value of the first plurality of weights to determine a limited value of the first weight; and use the limited value of the first weight to determine a second quantization factor for obtaining a second plurality of quantized weights associated with the corresponding layer of the fixed-point version of the machine learning model.

[0215] Example 11 includes a non-transitory computer-readable medium according to Example 1, wherein the floating-point version of the machine learning model is a floating-point version of a transformer network, the layer of the floating-point version of the machine learning model is a layer of the floating-point version of the transformer network, and the layer of the floating-point version of the transformer network corresponds to one of: (i) the output layer of a multilayer perceptron; (ii) a first-element-wise additive layer coupled to the output layer of the multilayer perceptron; or (iii) a second-element-wise additive layer coupled to the first-element-wise additive layer.

[0216] Example 12 includes an apparatus comprising an interface circuitry, machine-readable instructions, and at least one processor circuitry programmed based on the machine-readable instructions to: limit the values ​​of activations associated with layers of a floating-point version of a machine learning model to determine a limited value for the activations, the values ​​being based on calibration data applied to the floating-point version of the machine learning model; determine a quantization factor using the limited value for the activations to quantize activations associated with corresponding layers of a fixed-point version of the machine learning model; and configure the fixed-point version of the machine learning model on the apparatus using the quantization factor.

[0217] Example 13 includes a device according to Example 12, wherein one or more of the at least one processor circuitry will use the calibration data to initiate the execution of the floating-point version of the machine learning model, and cause the activated, throttled value to propagate to subsequent layers of the floating-point version of the machine learning model during the execution.

[0218] Example 14 includes the device according to Example 12, wherein the activation is a first activation, and one or more of the at least one processor circuitry will: observe values ​​of a plurality of activations associated with the layer of the floating-point version of the machine learning model, the values ​​of the plurality of activations being based on calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; determine a metric using the values ​​of the plurality of activations; and limit the value of the first activation using the metric.

[0219] Example 15 includes the device according to Example 14, wherein the metric is the standard deviation of the plurality of activated values, and one or more of the at least one processor circuitry are used to determine the mean of the plurality of activated values, and to limit the value of the first activated value by multiplying the mean and the standard deviation by a number.

[0220] Example 16 includes the device according to Example 12, wherein the activation is a first activation, the quantization factor includes a scaling factor and an offset factor, and one or more of the at least one processor circuitry determines the scaling factor and the offset factor by: determining a range of observations for a plurality of activations associated with the layer of the floating-point version of the machine learning model, the observations for the plurality of activations being based on calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; determining the scaling factor using the ratio of the range of observations to the quantization range associated with the corresponding layer of the fixed-point version of the machine learning model; and determining the offset factor using the ratio of the first observation among the observations to the scaling factor.

[0221] Example 17 includes a method comprising: limiting the values ​​of activations associated with a layer of a floating-point version of a machine learning model to determine a limited value for the activations, the values ​​of the activations being based on calibration data applied to the floating-point version of the machine learning model; using the limited value of the activations to determine a quantization factor to quantize the activations associated with a corresponding layer of a fixed-point version of the machine learning model; and using the quantization factor to configure the fixed-point version of the machine learning model on a device.

[0222] Example 18 includes the method according to Example 17, comprising initiating the execution of the floating-point version of the machine learning model using the calibration data, and causing the activated, throttled value to propagate to subsequent layers of the floating-point version of the machine learning model during the execution.

[0223] Example 19 includes the method according to Example 17, wherein the activation is a first activation, and the method includes: observing values ​​of a plurality of activations associated with the layer of the floating-point version of the machine learning model, the values ​​of the plurality of activations being based on calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; determining a metric using the values ​​of the plurality of activations; scaling the metric to determine a scaled metric; and limiting the value of the first activation using the scaled metric.

[0224] Example 20 includes the method according to Example 17, wherein the quantization factor is a first quantization factor, and the method includes: observing the values ​​of a first plurality of weights associated with a layer of the floating-point version of the machine learning model, the values ​​of the first plurality of weights being based on calibration data applied to the floating-point version of the machine learning model, the first plurality of weights corresponding to a single channel associated with the layer of the floating-point version of the machine learning model; using a metric to limit the value of a first weight among the first plurality of weights to determine a limited value of the first weight, the metric being based on the value of the first plurality of weights; and using the limited value of the first weight to determine a second quantization factor for quantizing a second plurality of weights associated with the corresponding layer of the fixed-point version of the machine learning model.

[0225] The appended claims are hereby incorporated by reference in this detailed description. While certain example systems, devices, articles of manufacture, and methods have been described herein, the scope of this patent is not limited thereto. Rather, this patent covers all systems, devices, articles of manufacture, and methods that fall fully within the scope of the claims of this patent.

Claims

1. A non-transitory computer-readable medium comprising computer-readable instructions to cause at least one processor circuit to at least: clip a value of an activation associated with a layer of a floating-point version of a machine learning model to determine a clipped value of the activation, the value of the activation based on calibration data applied to the floating-point version of the machine learning model; determine a quantization factor using the clipped value of the activation to quantize an activation associated with a corresponding layer of a fixed-point version of the machine learning model; and configure the fixed-point version of the machine learning model on a device using the quantization factor.

2. The non-transitory computer-readable medium of claim 1, wherein the instructions are to cause one or more of the at least one processor circuit to: initiate execution of the floating-point version of the machine learning model using the calibration data; and cause the clipped value of the activation to propagate to a subsequent layer of the floating-point version of the machine learning model during the execution.

3. The non-transitory computer-readable medium of claim 1, wherein the activation is a first activation, and the instructions are to cause one or more of the at least one processor circuit to: observe values of a plurality of activations associated with the layer of the floating-point version of the machine learning model, the values of the plurality of activations based on the calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; determine a metric using the values of the plurality of activations; and clip the value of the first activation using the metric.

4. The non-transitory computer-readable medium of claim 3, wherein the instructions are to cause one or more of the at least one processor circuit to: scale the metric to determine a scaled metric; and clip the value of the first activation using the scaled metric.

5. The non-transitory computer-readable medium of claim 3, wherein the metric is a standard deviation of the values of the plurality of activations.

6. The non-transitory computer-readable medium of claim 5, wherein the instructions are to cause one or more of the at least one processor circuit to: determine a mean of the values of the plurality of activations; and clip the value of the first activation using the mean and the standard deviation multiplied by a number.

7. The non-transitory computer-readable medium of claim 3, wherein the plurality of activations correspond to a single channel associated with the layer of the floating-point version of the machine learning model.

8. The non-transitory computer-readable medium of claim 1, wherein the activation is a first activation, the quantization factor includes a scaling factor, and the instructions are to cause one or more of the at least one processor circuit to determine the scaling factor by: determining an observed value range of a plurality of activations associated with the layer of the floating-point version of the machine learning model, the observed values of the plurality of activations based on the calibration data applied to the floating-point version of the machine learning model, the plurality of activations including the first activation; and ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determining the scaling factor using a ratio of the observed value range and a quantization range associated with the corresponding layer of the fixed-point version of the machine learning model.

9. The non-transitory computer-readable medium of claim 8, wherein the quantization factors include an offset factor, and the instructions will cause one or more of the at least one processor circuit to determine the offset factor using a ratio of a first observed value of the observed values and the scaling factor.

10. The non-transitory computer-readable medium of claim 1, wherein the quantization factor is a first quantization factor, and the instructions will cause one or more of the at least one processor circuit to: observe values of a first plurality of weights associated with the layer of the floating-point version of the machine learning model, the first plurality of weights corresponding to a single channel associated with the layer of the floating-point version of the machine learning model; clip values of first weights of the first plurality of weights using a metric to determine clipped values of the first weights, the metric based on the values of the first plurality of weights; and determine a second quantization factor to be used to obtain a second plurality of quantized weights associated with the corresponding layer of the fixed-point version of the machine learning model using the clipped values of the first weights.

11. The non-transitory computer-readable medium of claim 1, wherein the floating-point version of the machine learning model is a floating-point version of a transformer network, the layer of the floating-point version of the machine learning model is a layer of the floating-point version of the transformer network, and the layer of the floating-point version of the transformer network corresponds to one of: (i) an output layer of a multi-layer perceptron; (ii) a first-element-wise addition layer coupled to the output layer of the multi-layer perceptron; or (iii) a second-element-wise addition layer coupled to the first-element-wise addition layer.

12. An apparatus comprising: interface circuitry; machine-readable instructions; and at least one processor circuit programmed based on the machine-readable instructions to: clip a value of an activation associated with a layer of a floating-point version of a machine learning model to determine a clipped value of the activation, the value of the activation based on calibration data applied to the floating-point version of the machine learning model; determine a quantization factor using the clipped value of the activation to quantize an activation associated with a corresponding layer of a fixed-point version of the machine learning model; and configure the fixed-point version of the machine learning model on a device using the quantization factor.

13. The apparatus of claim 12, wherein one or more of the at least one processor circuit will: initiate execution of the floating-point version of the machine learning model using the calibration data; and cause the clipped value of the activation to propagate to a subsequent layer of the floating-point version of the machine learning model during the execution.

14. The apparatus of claim 12, wherein the activation is a first activation, and one or more of the at least one processor circuit will: observe values of a first plurality of weights associated with the layer of the floating-point version of the machine learning model, the first plurality of weights corresponding to a single channel associated with the layer of the floating-point version of the machine learning model; clip values of first weights of the first plurality of weights using a metric to determine clipped values of the first weights, the metric based on the values of the first plurality of weights; and determine a second quantization factor to be used to obtain a second plurality of quantized weights associated with the corresponding layer of the fixed-point version of the machine learning model using the clipped values of the first weights. observing values of a plurality of activations associated with the layer of the floating point version of the machine learning model, the values of the plurality of activations based on the calibration data applied to the floating point version of the machine learning model, the plurality of activations including the first activation; determining a metric using the values of the plurality of activations; and clipping the value of the first activation using the metric.

15. The device of claim 14, wherein the metric is a standard deviation of the values of the plurality of activations, and one or more of the at least one processor circuit is to: determine a mean of the values of the plurality of activations; and clip the value of the first activation using the mean and the standard deviation multiplied by a number.

16. The device of claim 12, wherein the activation is a first activation, the quantization factor includes a scale factor and an offset factor, and one or more of the at least one processor circuit is to determine the scale factor and the offset factor by: determining a range of observed values of a plurality of activations associated with the layer of the floating point version of the machine learning model, the observed values of the plurality of activations based on the calibration data applied to the floating point version of the machine learning model, the plurality of activations including the first activation; determining the scale factor using a ratio of the range of observed values and a quantization range associated with the corresponding layer of the fixed point version of the machine learning model; and determining the offset factor using a ratio of a first observed value of the observed values and the scale factor.

17. A method comprising: clipping a value of an activation associated with a layer of a floating point version of a machine learning model to determine a clipped value of the activation, the value of the activation based on calibration data applied to the floating point version of the machine learning model; determining a quantization factor using the clipped value of the activation to quantize an activation associated with a corresponding layer of a fixed point version of the machine learning model; and configuring the fixed point version of the machine learning model on a device using the quantization factor.

18. The method of claim 17, including: initiating execution of the floating point version of the machine learning model using the calibration data; and causing the clipped value of the activation to propagate to a subsequent layer of the floating point version of the machine learning model during the execution.

19. The method of claim 17, wherein the activation is a first activation, and the method includes: observing values of a plurality of activations associated with the layer of the floating point version of the machine learning model, the values of the plurality of activations based on the calibration data applied to the floating point version of the machine learning model, the plurality of activations including the first activation; determining a metric using the values of the plurality of activations; scaling the metric to determine a scaled metric; and clipping the value of the first activation using the scaled metric.

20. The method of claim 17, wherein the quantization factor is a first quantization factor, and the method includes: observing values of a first plurality of weights associated with the layer of the floating-point version of the machine learning model, the values of the first plurality of weights based on the calibration data applied to the floating-point version of the machine learning model, the first plurality of weights corresponding to a single channel associated with the layer of the floating-point version of the machine learning model; clipping a value of a first weight of the first plurality of weights using a metric to determine a clipped value of the first weight, the metric based on the values of the first plurality of weights; and determining a second quantization factor for quantizing a second plurality of weights associated with the corresponding layer of the fixed-point version of the machine learning model using the clipped value of the first weight.

Images