Quantized neural network architecture

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for quantized machine learning. A quantized input matrix is accessed at a layer of a neural network, and a first intermediate value is generated in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator. The first intermediate value is normalized based at least in part on one or more preamble bit (s) of the first intermediate value, and the normalized first intermediate value is inverse quantized. A second intermediate value is generated by applying a rounding right-shift operation to the inversely quantized normalized first intermediate value, and activation data is generated by applying an activation function to the second intermediate value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application Ser. No. 17 / 934,476, filed on Sept. 22, 2022, which is hereby incorporated by reference herein.

[0003] introduction

[0004] Aspects of the present disclosure relate to quantized machine learning.

[0005] Machine learning (such as using neural network models) is a general and dynamic tool that helps solve a wide variety of tasks, including computer vision, speech recognition, signal analysis, etc. However, such models generally require a large amount of computing resources during training and during inference. For example, the parameters of the model (e.g., weights and / or biases) and input and activation data (when input samples are processed by the model) are usually stored and processed in floating point format, which not only requires a large amount of memory space, but also introduces a large amount of data transmission and processing overhead.

[0006] Some conventional approaches to reducing the computational burden of such models involve quantization of some or all of the data. Quantization generally refers to the process of converting or constraining inputs from a relatively large value space (e.g., a continuous set of values) to a relatively small value space (e.g., a discrete set of values), thereby reducing memory, transmission, and processing overhead. For example, affine integer quantization uses a mix of integer and floating-point computations to improve latency and memory requirements for training and inference of machine learning models.

[0007] Therefore, techniques for improved quantitative machine learning are needed. Summary of the invention

[0008] One aspect provides a method, comprising: accessing a quantized input matrix at a layer of a neural network; generating a first intermediate value in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using an accumulator; normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; dequantizing the normalized first intermediate value; generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; and generating activation data by applying an activation function to the second intermediate value.

[0009] Other aspects provide processing systems configured to perform the aforementioned methods and those described herein; non-transitory computer-readable media comprising instructions that, when executed by a processor of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0010] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0012] Figure 1 Depicts an example architecture for improved quantized machine learning.

[0013] Figure 2A Depicted is a piecewise linear approximation of the TanH activation function for improved quantized machine learning.

[0014] Figure 2B Depicts a piecewise linear approximation of the sigmoid activation function for improved quantized machine learning.

[0015] Figure 3 An example flow chart illustrating a method for processing data using quantitative machine learning is depicted.

[0016] Figure 4 An example flow chart illustrating a method for generating a piecewise linear approximation of an activation function for improved machine learning is depicted.

[0017] Figure 5 An example flow chart illustrating a method for quantifying machine learning is depicted.

[0018] Figure 6 An example processing system configured to perform various aspects of the present disclosure is depicted.

[0019] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0020] Aspects of the present disclosure provide improved quantized neural network architectures.

[0021] Neural network quantization using affine integer quantization is a useful paradigm for computing inference workloads on edge devices (e.g., devices with relatively limited computing resources such as smartphones). In some such approaches, fixed-point integer representations can be used to represent floating-point quantities in the data path to avoid or reduce the need for dedicated floating-point hardware units. However, selecting an appropriate fixed-point format to represent the various variables in the data path is challenging, and the challenge of maintaining fractional precision is significantly exacerbated when small bit widths (e.g., less than 32 bits) are desired for output and intermediate variables.

[0022] In some aspects of the present disclosure, new and / or modified data paths are provided to enable maximum fractional precision to be maintained while minimizing the bit width required to store variables in memory. The present disclosure provides a reproducible framework for designing optimized training and inference workloads, and uses operations and / or hardware that are already available on a wide variety of processor architectures. Therefore, the various aspects described herein can be beneficially implemented in a wide range of machine learning architectures.

[0023] Aspects of the present disclosure can be readily applied to the computation of any neural network training or inference workload. In some aspects, the architectures and techniques disclosed herein can be particularly useful on resource-constrained platforms (e.g., platforms with limited memory, bandwidth, computing power, etc.) because the disclosed architectures and techniques enable accurate training and inference with substantially reduced computational overhead through improved quantized data paths. For example, aspects of the present disclosure enable the use of quantized neural networks on a variety of devices such as wearable devices with on-chip inference capabilities, smart devices (including smart home appliances), Internet of Things (IoT) devices, edge processing devices, always-on devices, etc.

[0024] In various aspects of the present disclosure, selective normalization and rounding right-shift operations are used to modify the network data path in a manner that achieves improved fractional precision (resulting in prediction accuracy that is the same as or better than conventional quantization systems) and reduced computational cost (e.g., allowing operations to be performed using smaller bit widths).

[0025] As used herein, a fixed-point value is represented or defined as N[S / U]F, where N is the bit width of the value, S / U refers to whether the value is signed, and F refers to the number of fraction bits. For example, a 16S12 fixed-point number is a number with 12 fraction bits (e.g., with a range of [-2 3 ,2 3 -2 -12 ]) is a 16-bit signed number.

[0026] In various aspects of the present disclosure, a "right shift" operation refers to a rounding right shift. A rounding right shift generally involves determining and adding a rounding bias when shifting the value, thereby improving the propagation of fractional values. For example, in at least one aspect, to shift a value a right by n bits, the system may calculate the rounding bias as 1<<(n-1) (e.g., the value one is shifted left by n-1), and then calculate the output of the rounding right shift as (a+rounding_bias)>>n) (e.g., the sum of the rounding bias and a is shifted right by n). If n is less than or equal to zero, the system may instead generate the output as sat(a<<(-n)) (e.g., a is shifted left by (-n) and saturated to the full bit width available).

[0027] In various aspects of the present disclosure, a "normalization" operation returns the number of leading sign bits of an input value and a normalized value of the number of leading sign bits shifted left. For example, a normalized value of an input a can be calculated by counting the number of leading sign bits of a (e.g., the number of most significant bits that are the same as the sign bit) and shifting a left by a determined number of leading sign bits. This normalization can improve the accuracy of the model by retaining lower order bits that would otherwise be lost during subsequent truncation (e.g., using saturation and / or right shift operations). That is, truncation or right shifting is likely to only discard the bits added by the normalization operation, rather than the bits that carry the actual data, as discussed in more detail below. This allows the normalized and / or right shifted values ​​to be stored with significantly smaller bit widths, as compared to conventional systems.

[0028] By using normalization and rounding right shift operations in the data path, aspects of the present disclosure can provide accurate and reliable predictions for model parameters and intermediate values ​​in the model with significantly reduced bit widths. For example, rather than relying on 32-bit wide numbers for intermediate values, some aspects of the present disclosure can use 16 bits without losing model accuracy (or while minimizing the loss of model accuracy).

[0029] Example architecture for improved quantized machine learning

[0030] Figure 1 An example architecture 100 for improved quantized machine learning is depicted. In the illustrated example, the architecture 100 corresponds to a data path and a set of operations for implementing a fully connected layer of a neural network.

[0031] In some aspects, to quantize a given value, the value is divided by a learned scaling factor, and a learned zero value is added to the result. In general, the scaling factor can be learned based on a range of values ​​found in unquantized data during training, while the zero point can be established based on finding a linear relationship between the minimum quantized value and the minimum unquantized value found during training. For example, the input tensor X can be quantized as Where X qis the quantized input (e.g., the quantized version of the input tensor X), S X is the scaling factor learned for the input tensor X, and Z X are the learned zeros of the input tensor X.

[0032] In some aspects discussed herein, a "zero point" may be generally used to offset the range of input data and may correspond to a quantized value of an input value of 0. For example, if the zero point is 0, then an input value of 0 remains 0 after quantization, whereas if the zero point is 5, then an input value of 0 becomes a value of 5 after quantization. A scaling factor may be generally used to define the range of quantized data (e.g., to reduce or increase the range of an input).

[0033] In some aspects, to dequantize a value, the learned zero point is subtracted from the value and the difference is multiplied by the learned scaling factor. q Can be dequantized to X = S X (X q -Z X ).

[0034] In the illustrated example, the output of the fully connected layer of the neural network can be defined as Y = f act (WX+b), where Y is the output tensor, X is the input tensor, W is the weight of the layer, b is the bias of the layer, and f act Corresponding to the application of an activation function such as ReLU, etc. In the illustrated example using quantized values, the quantized input, weight, and bias may be given as X q , W q and b q Thus, in one aspect, the output of a quantized fully connected layer may be defined using the following equation (1), where dequant() is a dequantization operation.

[0035] Y=f act (dequant(W q )·dequant(X q )+dequant(b q )) (1)

[0036] As discussed above, the inverse quantization operation may be performed using the scaling factor and the zero points learned for each value. Thus, the output of the fully connected quantization layer may be defined using the following equation (2).

[0037] Y=f act (S W (W q -Z W )·S X (X q -Z X )+S b (bq -Z b )) (2)

[0038] In some aspects, equation (2) can be rearranged to provide Figure 1 The illustrated set of operations or blocks that implement optimized data processing. Specifically, in the illustrated example, the output Y can be defined using the following equation (3).

[0039]

[0040] In at least one aspect, equation (3) can be rewritten again with new variable names for ease of reference using equation (4) below Figure 1 In equation (4), D = S W S X , M=W q X q -Z W X q and

[0041]

[0042] Y=f act (D(MA)) (4)

[0043] The illustrated architecture 100 is a quantized fully connected layer of a neural network. For example, block 102 may be performed using an accumulator (e.g., storing and accumulating individual results or portions of the matrix multiplication operation 108 as the results are calculated iteratively or sequentially), thereby implementing the MA portion of equation (4). Specifically, A may be referred to as an accumulator initializer, which may be pre-computed and used to initialize the accumulator. That is, since each value in A is known when training is complete, a may be calculated offline and used to initialize the accumulator. As used herein, initializing an accumulator refers to setting the (initial) value of the accumulator to a specific value (e.g., A) and using the initialized value as a parameter for subsequent operations performed using the accumulator (e.g., for W q and X q In addition, the M part (which may be called a matrix multiplication and may be performed during runtime using integers rather than floating point values) may be computed using a (pre-initialized) accumulator.

[0044] In the illustrated example, this block 102 of the architecture 100 includes converting the quantized input (X q )104 and quantization weight (W q ) is provided to the matrix multiplication operation 108 (e.g., performed using a pre-initialized accumulator, as discussed above).

[0045] As shown, the resulting value in the accumulator may then be normalized (e.g., using the normalization operation discussed above, where the value is left-shifted by the number of leading sign bits in the value) to produce a normalized tensor 110. As discussed above, this left-shift normalization operation fills the lower order bits with values ​​that do not carry real data (e.g., zeros or ones filled without carrying real information). Thus, during a subsequent right shift or saturation operation, these lower order values ​​may be discarded without losing the real information now stored in the higher order bits. This significantly improves the fractional accuracy of the architecture 100.

[0046] In some aspects, the normalization operation may maintain the normalized tensor 110 in the same bit width as the original output of the accumulator. For example, in some conventional systems, the output of the accumulator is 32 bits. In one aspect, the normalization operation may maintain the 32-bit width, wherein the value actually carrying the data is shifted left by the number of leading sign bits. In one such aspect, the normalized tensor 110 may be specified as 32S0, thereby indicating that the tensor is a signed 32-bit integer with a zero fractional bit. Although not depicted in the illustrated example, in one aspect, the system may additionally keep tracking the number of leading sign bits for downstream operations. If the number of leading sign bits (which corresponds to the number of bits that the normalized tensor 110 is shifted left) is not tracked, the actual real output of block 102 (e.g., the output of the accumulator) may be lost. Typically, tracking the number of leading sign bits can be performed by storing the value in a register or other memory. The number of leading sign bits may then be provided as an input to downstream operations (e.g., to determine the appropriate shift amount before applying the activation function, as discussed in more detail below).

[0047] In the illustrated architecture, the normalized tensor 110 is then provided to block 112, which applies an inverse quantization scale (e.g., computes D, which may be referred to as an inverse quantization operation) to the weight tensor and the input tensor. Specifically, the weight scaling factor (S W )114 and input scaling factor (S X ) 116 is provided to a multiplication operation 118 along with the normalized tensor 110. This results in an intermediate tensor 120 that is provided as an input to an activation function 124. Block 112 may thus be used to complete the portion of equation (4) that is provided as an input to the activation function.

[0048] In some aspects, block 112 involves floating point multiplication. In at least one aspect, the system performs block 112 by converting each of the inverse quantized scaling coefficients 114 and 116 into a corresponding mantissa (e.g., in 16S15) with an integer exponent. That is, the system may decompose or depict each of the floating point values ​​into a fixed point mantissa and a corresponding exponent, thereby allowing them (along with the normalized tensor 110) to be multiplied using a fixed point or integer multiplication operation 118 instead of a floating point calculation.

[0049] In some aspects, the multiplication operation also receives a determined number of leading sign bits for normalizing the normalized tensor 110. For example, the number of leading sign bits can be provided as another exponent in the multiplication (e.g., 2 N , where N is the number of leading sign bits), and exponents determined for the inverse quantized scaling factors 114 and 116. This ensures that the resulting tensor 120 has the correct values ​​(e.g., as if no normalization was performed).

[0050] In some aspects, the tensor 120 is first right-shifted (e.g., using a rounding right shift) before being provided to the activation function 124. For example, since the multiplication operation 118 may generally result in a larger bit-width value (e.g., transforming a 32-bit input to a 64-bit output), the value is generally right-shifted back to a lower value. In some aspects, the right-shift process is performed at least in part based on the number of leading sign bits determined above, as discussed in more detail below. In many conventional systems, the tensor 120 has the same bit width as the output of the accumulator. For example, if the accumulator tensor is 32 bits, conventional systems generally also right-shift the tensor 120 back to 32 bits. As discussed above, this results in a loss of fractional precision because multiple bits carrying useful information are discarded.

[0051] However, since the normalized tensor 110 is first left-shifted prior to block 112, the result of the multiplication operation 118 can be right-shifted to a lower bit width, thereby discarding only bits that do not carry real information (e.g., erroneous bits). In at least one aspect, rather than shifting back to the size of the normalized tensor 110 (e.g., 32 bits), block 112 can involve right-shifting to a smaller bit width. For example, in one such aspect, the tensor 120 can be right-shifted to a size that is half the bit width of the normalized tensor 110 (e.g., right-shifted to 16 bits).

[0052] In at least one aspect, tensor 120 (e.g., the output of the accumulator) is right-shifted by different amounts based, at least in part, on the specific characteristics of the activation function used. That is, the input to activation function 124 (e.g., tensor 120) can be stored or represented using different formats (e.g., 16S13 or 16S12) based on the characteristics of the function. For example, as discussed in more detail below, in some aspects, the input to the hyperbolic tangent (TanH) activation function can be efficiently stored in 16S13 (e.g., a signed 16-bit value with 13 fractional bits), while the input to the sigmoid activation function can be efficiently stored in 16S12 (e.g., a signed 16-bit value with 12 fractional bits).

[0053] In some aspects, the number of leading sign bits (determined during the normalization step above) can be used to determine the net right shift required to convert the tensor 120 to the format expected as input to the activation function 124. For example, with respect to the sigmoid activation function (which may receive 16S12 values), the shift amount n may be defined as n=-k+sb+12, where k is an exponent used to represent the floating point quantization scale in fixed point, and sb is the number of leading sign bits previously determined. Similarly, with respect to the TanH activation function (which may receive 16S13 values), the shift amount n may be defined as n=-k+sb+13.

[0054] By right-shifting to a smaller bit width than that used by conventional systems, architecture 100 achieves significantly improved computational latency and reduced computational cost. This enables machine learning to be performed on a wider array of devices, including smartphones and other edge devices and other resource-constrained devices, as discussed above. Additionally, fractional precision (and therefore model prediction accuracy) is maintained because the normalization operation allows the bits that carry information to be preserved after the subsequent right shift.

[0055] As shown, the tensor 120 is then processed by an activation function 124. The activation function 124 may generally correspond to any function, including nonlinear functions such as rectified linear units (ReLU), sigmoid, hyperbolic tangent (TanH), etc. While some simple activation functions can be performed with little computational cost (e.g., ReLU), other simple activation functions have greater complexity (e.g., sigmoid and TanH), which conventionally require more complex processing or calculations. For example, the sigmoid activation function may be defined as This requires calculating the exponential. Similarly, the TanH function can be defined as This also requires calculating the index value.

[0056] In the illustrated example, because the tensor 120 can be a smaller bit width than the output of the accumulator (e.g., smaller than the normalized tensor 110), the tensor 120 can generally be processed by the activation function 124 more efficiently than conventional systems. For example, computing the activation function 124 on a 16-bit input tensor 120 significantly reduces computational and memory overhead, as compared to a conventional 32-bit input. In the illustrated example, the resulting output (Y) 126 (also referred to as an activation tensor in some aspects) can similarly have the same bit width of the tensor 120 (e.g., 16 bits), which is smaller than the normalized tensor 110 and smaller than conventional quantization systems.

[0057] In some aspects, to further reduce the computational cost of applying the activation function 124, the system may generate (e.g., offline) and use a piecewise linear approximation of the activation function. For example, as discussed in more detail below, the dynamic range of the activation function may be identified, where the dynamic range corresponds to a region or range of input values ​​where the output of the function does not change or is within a threshold distance of a fixed value. For example, outside the range [-4, 4], the TanH function outputs values ​​very close to -1 and 1, respectively. Therefore, in one aspect, the TanH function can be approximated by clamping all input values ​​less than -4 to an output of -1, and clamping all values ​​greater than 4 to an output value of 1. This can significantly reduce the computational overhead of applying the TanH function without losing significant accuracy.

[0058] In some aspects, the identified dynamic range of the activation function may thereafter be delineated or divided into a set of segments or portions offline, and a linear approximation may be generated for each such segment or portion. For example, the dynamic range may be divided into 16 or 32 discrete segments. During runtime, the application of the activation function 124 may be performed using the piecewise linear approximation. Such a linear function significantly reduces computational cost, as compared to conventional methods, without significant loss of accuracy.

[0059] In some aspects, output 126 may be output from the layer. For example, if architecture 100 is in the last layer of a neural network, output 126 may be used as output from the model. In the illustrated aspect, output 126 may additionally be quantized. This may allow output 126 to be used as quantized input to subsequent layers of the model, and / or as quantized output from the model.

[0060] In the illustrated example, block 128 is used to quantize output 126 to generate a quantized output (Y q ) 140. As shown, the system can use the multiplication operation 132 to convert the learned scaling factor (S Y ) 130 and output 126 to do so. That is, the system may multiply output 126 by scaling factor 130 or the inverse of scaling factor 130 (which may be learned based on output values ​​observed during training, as discussed above). In some aspects, multiplication operation 132 may be similarly performed using the mantissa operation described above (e.g., by decomposing scaling factor 130 into a 16S15 mantissa with an integer exponent) to improve the efficiency of the operation.

[0061] In some aspects, the resulting output of the multiplication operation 132 is right-shifted (e.g., using a rounding right shift) to the same bit width as the original input (e.g., as the output 126 of the activation function). For example, the tensor 134 can be right-shifted (also referred to as saturated) to 16 bits.

[0062] The resulting tensor 134 can then be combined with the learned zero point (Z Y ) 136 (the learned zero point is also learned based on the output values ​​observed during training). The resulting quantized output 140 can then be output from the layer (e.g., as a quantized input to a subsequent layer, or as a quantized output from a model). In some aspects, the quantized output 140 has a smaller bit width than the output 126 (and can have the same bit width as the quantized input 104 and / or the quantized weight 106), thereby improving the efficiency of the machine learning model. For example, the quantized output 140 can have a bit width of 8.

[0063] Example piecewise linear approximation of activation functions for improved quantized machine learning

[0064] Figure 2A Graph 200A depicts an example piecewise linear approximation 205 for improved quantized machine learning. Specifically, piecewise linear approximation 205 corresponds to the TanH activation function. In some aspects, piecewise linear approximation 205 may be generated offline (e.g., before training begins). Approximation 205 may then be used during training and during runtime (e.g., in place of Figure 1 The activation function 124 of the model is used to reduce the computational overhead of the model. In some aspects, the piecewise linear approximation 205 can be stored in a memory such as a lookup table for efficient access.

[0065] As depicted, the dynamic range 208 of the TanH function has been identified as having a lower limit indicated by line 210 and an upper limit indicated by line 220. In some aspects, the limits of the dynamic range 208 can be determined by finding an input value (e.g., an x ​​value in the illustrated example) beyond which the corresponding output does not change beyond a defined threshold (or within a defined distance of a fixed value). For example, below a value indicated by line 210 (which may correspond to an input value of -4), the output of TanH is within a configurable threshold distance from a value of -1 indicated by line 215. Similarly, above a value indicated by line 220 (which may correspond to an input value of 4), the output of TanH is within a configurable threshold distance from a value of 1 indicated by line 225.

[0066] Thus, in some aspects, the piecewise linear approximation 205 is generated by using fixed output values ​​for input values ​​outside the dynamic range 208 (e.g., a value of -1 for inputs less than or equal to -4, and a value of 1 for inputs greater than or equal to 4). This can significantly reduce the computational overhead of the function because values ​​outside the dynamic range 208 are clamped to fixed values.

[0067] Furthermore, in the illustrated example, the dynamic range 208 is divided or depicted into a set of discrete segments or portions, such as 230A and 230B (collectively, segments 230). Although two such segments 230A and 230B are indicated for clarity, there may be any number of such segments in the dynamic range 208. As discussed above, the system may generate a less complex approximation of the original activation function within each depicted segment 230 of the dynamic range 208. For example, in the illustrated aspect, each respective segment 230 is individually approximated by fitting a respective linear function with a respective slope and bias or offset value. Such linear approximations may be evaluated significantly faster than conventional TanH functions (as well as more efficiently and faster than conventional approximations that generally rely on higher order functions (such as cubic functions)).

[0068] In various aspects, the dynamic range 208 may generally be divided into any number of segments 230. In some aspects, the dynamic range 208 may be divided into segments 230 of equal or different sizes. For example, in one such aspect, for regions where the output value of the activation function changes rapidly or significantly, the segments 230 may be narrower or smaller (e.g., covering a smaller range of inputs), and for regions where the activation function changes less significantly, the segments may be wider. In some aspects, the number and / or size of the segments may be selected (e.g., by an administrator or user) based on factors such as the specific activation function being approximated, the hardware on which the piecewise linear approximation 205 will be executed, etc. In at least one aspect, the dynamic range 208 of the piecewise linear approximation 205 is divided into 32 segments 230. In some aspects, the entire piecewise linear approximation 205 is depicted as 32 segments 230 (e.g., thirty segments within the dynamic range 208, one segment for values ​​less than the threshold represented by line 210, and one segment for values ​​greater than the threshold represented by line 220).

[0069] By using a piecewise linear approximation 205 with a limited dynamic range 208 and a fixed output beyond the dynamic range 208, the system is able to reduce the latency and expense of applying the activation function. Although the illustrated example depicts a piecewise linear approximation 205 of a TanH function, aspects of the present disclosure may be readily adapted to generate a piecewise linear approximation for any activation function.

[0070] Figure 2B A graph 200B depicts an example piecewise linear approximation 250 for improved quantized machine learning. Specifically, the piecewise linear approximation 250 corresponds to a sigmoid activation function. Figure 2A As discussed above, in some aspects, piecewise linear approximation 250 may be generated offline (e.g., before training begins). Approximation 250 may then be used during training and during runtime (e.g., instead of Figure 1The activation function 124 of the example can be used to reduce the computational overhead of the model. In some aspects, the piecewise linear approximation 250 can be stored in a memory (such as a lookup table) for efficient access.

[0071] As depicted, the dynamic range 253 of the sigmoid function has been identified as having a lower limit indicated by line 255 and an upper limit indicated by line 265. In some aspects, the limits of the dynamic range 253 can be determined by finding input values ​​(e.g., x values ​​in the illustrated example) beyond which the corresponding output does not change beyond a defined threshold (or within a defined distance of a fixed value). For example, below the value indicated by line 255 (which may correspond to an input value of -8), the output of the sigmoid is indicated by line 260 to be within a configurable threshold distance from a value of 0. Similarly, above the value indicated by line 265 (which may correspond to an input value of 8), the output of the sigmoid is indicated by line 270 to be within a configurable threshold distance from a value of 1.

[0072] Thus, in some aspects, piecewise linear approximation 250 is generated by using fixed output values ​​for input values ​​outside of dynamic range 253 (e.g., a value of 0 for inputs less than or equal to -8, and a value of 1 for inputs greater than or equal to 8). This can significantly reduce the computational overhead of the function because values ​​outside of dynamic range 253 are clamped to fixed values.

[0073] Furthermore, in the illustrated example, the dynamic range 253 is divided or depicted into a set of discrete portions or segments 275A and 275B (collectively, segments 275). Although two such segments 275A and 275B are indicated for clarity, there may be any number of such segments in the dynamic range 253. As discussed above, the system may generate a less complex approximation of the original activation function within each depicted segment 275 of the dynamic range 253. For example, in the illustrated aspect, each respective segment 275 is individually approximated by fitting a respective linear function with a respective slope and bias or offset value. Such linear approximations may be evaluated significantly faster than conventional sigmoid functions (as well as more efficiently and faster than conventional approximations that generally rely on higher order functions such as cubic functions).

[0074] In various aspects, the dynamic range 253 may generally be divided into any number of segments 275. In some aspects, the number of segments may be selected (e.g., by an administrator or user) based on factors such as, for example, the specific activation function being approximated, the hardware on which the piecewise linear approximation 250 will be executed, etc. In at least one aspect, the dynamic range 253 of the piecewise linear approximation 250 is divided into 32 segments 275. In some aspects, the entire piecewise linear approximation 250 is depicted as 32 segments 275 (e.g., thirty segments within the dynamic range 253, one segment for values ​​less than a threshold represented by line 255, and one segment for values ​​greater than a threshold represented by line 265).

[0075] By using a piecewise linear approximation 250 with a limited dynamic range 253 and a fixed output beyond the dynamic range 253, the system is able to significantly reduce the latency and expense of applying the activation function. Although the illustrated example depicts a piecewise linear approximation 250 of a sigmoid function, aspects of the present disclosure may be readily adapted to generate a piecewise linear approximation for any activation function.

[0076] Example method for processing data using quantized machine learning

[0077] Figure 3 An example flow chart illustrating a method 300 for processing data using quantitative machine learning is depicted. In some aspects, the method 300 is directed to Figure 1 In one aspect, method 300 may be performed by a machine learning system (such as the one described below with reference to Figure 6 The system discussed in more detail herein is executed by the processor and can be used during training (e.g., during a forward pass of data through the network) or during inference.

[0078] At block 305, the machine learning system initializes the accumulators based on precomputed values. In some aspects, the precomputed values ​​are generated based at least in part on the quantization statistics learned during training, based on the values ​​seen so far for each tensor (e.g., for the input tensors and / or the weight tensors). For example, as discussed above, the accumulators may be initialized based on: the quantized weights (W q ), the zero point of the input tensor (Z X ), the zero point of the weight tensor (Z W ), bias terms or tensor scaling factors (S b ), the scaling factor of the weight tensor (S W ), the scaling factor of the input tensor (S X ), quantization deviation (b q ) and / or the zero point of the offset (Z b ).

[0079] For example, as discussed above with reference to equation (3), the machine learning system may calculate And use this value to initialize the accumulator. Since each of these terms is known once training is complete (e.g., quantized weights and biases, and quantized statistics for weights, biases, and inputs), the machine learning system can pre-compute this value offline and does not need to do so during runtime inference.

[0080] At block 310, the machine learning system may use the initialized accumulator to convert the quantized input tensor X q (For example, Figure 1 quantized input 104) and quantized weight tensor W q (For example, Figure 1 The quantization weight 106) is multiplied.

[0081] At block 315, the machine learning system then normalizes the value stored in the accumulator (e.g., the result of multiplying the quantized input by the quantized weight plus the pre-computed accumulator initialization value). In some aspects, as discussed above, the normalization operation involves determining the number of leading sign bits on the value and shifting the value left by that number of leading sign bits. This can help preserve bits that carry information during subsequent operations.

[0082] At block 320, the machine learning system may apply de-quantization scaling of the weights and inputs, as discussed above (e.g., with reference to Figure 1 In some aspects, as discussed above, the machine learning system can perform this dequantization more efficiently and with reduced computational cost by decomposing each value into a fixed-point mantissa and a corresponding exponent and multiplying the mantissa / exponent (including the number of leading sign bits determined during the normalization operation).

[0083] At block 325, the machine learning system then right-shifts the resulting tensor (generated in block 320). In the illustrated example, the machine learning system right-shifts the data based at least in part on the activation function to be used. That is, in one such aspect, different formats (e.g., 16S13 or 16S12) may be used to store or represent the input to the activation function based on the characteristics of the function (e.g., based on the width of the dynamic range of the piecewise linear approximation). For example, a piecewise linear approximation to a TanH function (e.g., Figure 2A The input to the piecewise linear approximation 205 of ) can be efficiently stored in 16S13 (e.g., a signed 16-bit value with 13 fractional bits). As another example, the piecewise linear approximation to the sigmoid function (e.g., Figure 2B The input to the piecewise linear approximation 250) can be efficiently stored in 16S12 (e.g., a signed 16-bit value with 12 fraction bits).

[0084] In some aspects, as discussed above, the right shift operation is performed based at least in part on the number of leading sign bits determined in block 315. That is, the magnitude of the right shift may be defined based at least in part on the number of leading sign bits determined during normalization.

[0085] At block 330, the machine learning system applies an activation function (or applies a piecewise linear approximation of an activation function), as discussed above. That is, the machine learning system may apply the activation function by providing the tensor generated in block 325 to the activation function or piecewise linear approximation (such as Figure 2A Piecewise linear approximation of 205 or Figure 2B 250) to calculate the activation output Y (e.g., Figure 1 126). Since the input to the activation function is generally stored in a smaller bit width than conventional systems (e.g., stored in half the bit width of the accumulator output, whereas conventional systems keep the activation input at the same bit width as the accumulator output), block 330 can be performed with significantly reduced computational overhead. Additionally, if a piecewise linear approximation is used, the cost is even further reduced.

[0086] At block 335, the machine learning system can optionally quantize the activation data (e.g., the output of the activation function). In some aspects, rather than quantizing the activation data, the activation data can be provided as an output from the neural network (e.g., provided to a downstream processing task), or as an unquantized input to a downstream layer of the model. In other aspects, the machine learning system can quantize the activation data (e.g., as described above with reference to Figure 1 The quantized data may then be provided as quantized input to subsequent layers of the model, and / or as quantized output from the model.

[0087] As discussed above, the method 300 generally achieves reduced computational overhead (eg, reduced latency, reduced power consumption, reduced computational or processing expense, reduced heat generation, etc.) compared to conventional methods.

[0088] It should be noted that Figure 3 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0089] Example method for generating piecewise linear approximations of activation functions for improved machine learning

[0090] Figure 4 An example flow chart illustrating a method 400 for generating a piecewise linear approximation of an activation function for improved machine learning is depicted. In some aspects, the method 400 provides a method for generating Figure 2A Piecewise linear approximation 205 and Figure 2BIn one aspect, the method 400 may be performed by a machine learning system such as the one described below with reference to Figure 6 The systems discussed in greater detail are performed offline (e.g., prior to training a machine learning model and / or performing inference using the trained model).

[0091] At block 405, the machine learning system identifies an activation function to use. In some aspects, a user or administrator may specify which activation function to use for each layer of the neural network. In some aspects, the activation function may be identified in a predefined model architecture.

[0092] At block 410, the machine learning system determines a dynamic range for the identified activation function. As discussed above, in some aspects, the dynamic range may correspond to a range of input values ​​for which the output value changes above a certain threshold. In some aspects, the dynamic range is determined by identifying a range of input values ​​where, outside the dynamic range, the output value changes below a certain threshold and / or is within a defined distance of a fixed value (or where the values ​​outside the dynamic range are sufficiently close to a linear function). For example, as discussed above, the TanH function may have a dynamic range of [-4, 4] (as discussed above with reference to Figure 2A discussed), while the sigmoid function has a dynamic range of [-8,8] (as discussed above Figure 2B In one aspect, input values ​​outside the dynamic range can be clamped to a fixed value.

[0093] At block 415, the machine learning system divides the determined dynamic range into a set of discrete segments or portions. Figure 2A and 2B As discussed, the machine learning system may characterize the dynamic range of a function into 32 equal width sections (e.g., Figure 2A Fragment 230 and / or Figure 2B Fragment 275).

[0094] At block 420, the machine learning system selects one of the segments in the dynamic range. In aspects, the machine learning system may select the segment using any suitable criteria (including randomly or pseudo-randomly), as each segment will be evaluated in turn during method 400. Although an iterative process is depicted for conceptual clarity (selecting and approximating each segment in turn), in some aspects, the machine learning system may select and process some or all of the segments in parallel.

[0095] At block 425, the machine learning system generates a linear approximation of the activation function relative to the selected segment. For example, the machine learning system can fit a linear function (with a slope and a bias) to the activation function values ​​within the segment. Typically, evaluating the linear approximation requires significantly reduced computational overhead, as compared to evaluating the original activation function. Additionally, depending on the size of the segment (e.g., the number of segments used to partition the dynamic range), the linear approximation can maintain a high degree of accuracy.

[0096] At block 430, the machine learning system determines whether there are any additional segments in the dynamic range that have not yet been approximated. If so, the method 400 returns to block 420. If not, the method 400 continues to block 435.

[0097] At block 435, the machine learning system can store the generated piecewise linear approximation of the activation function for future use. For example, the machine learning system can store a set of linear functions (e.g., slopes and biases) that make up the dynamic range (and in some aspects, portions outside the dynamic range), along with an indication of the range of input values ​​corresponding to each linear function (e.g., indicating which function should be used to generate the output given a particular input value). In some aspects, the machine learning system stores the piecewise linear approximation as part of the machine learning model (e.g., in memory, using a lookup table, etc.) so that the approximation can be used during training and / or inference.

[0098] As discussed above, piecewise linear approximation of activation functions can substantially reduce the computational overhead of applying activation functions, thereby significantly reducing training and inference latency, as well as reducing other burdens including power consumption.

[0099] It should be noted that Figure 4 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0100] Example Methods for Quantized Machine Learning

[0101] Figure 5 An example flow chart illustrating a method 500 for quantizing machine learning is depicted.

[0102] At block 505, a quantized input matrix is ​​accessed at a layer of a neural network.

[0103] At block 510 , a first intermediate value is generated in an accumulator by performing a matrix multiplication of a quantized input matrix and a quantized weight matrix associated with a layer of a neural network using the accumulator.

[0104] At block 515 , the first intermediate value is normalized based at least in part on one or more leading sign bits of the first intermediate value.

[0105] At block 520, the normalized first intermediate value is dequantized.

[0106] At block 525 , a second intermediate value is generated by applying a round right shift operation to the dequantized normalized first intermediate value.

[0107] At block 530 , activation data is generated by applying an activation function to the second intermediate value.

[0108] In some aspects, normalizing the first intermediate value includes: determining a number of leading sign bits of the first intermediate value; and left shifting the first intermediate value by the determined number of leading sign bits.

[0109] In some aspects, the second intermediate value has a smaller bit width than the normalized first intermediate value.

[0110] In some aspects, the normalized first intermediate value has a bit width that is at least twice the bit width of the second intermediate value.

[0111] In some aspects, wherein: the normalized first intermediate value has a bit width of 32 bits; and the second intermediate value has a bit width of 16 bits.

[0112] In some aspects, applying the activation function includes applying a piecewise linear approximation of the activation function.

[0113] In some aspects, a piecewise linear approximation of an activation function is generated by: identifying a dynamic range of the activation function, where outside the dynamic range, values ​​of the activation function are within a threshold distance from one or more fixed values; dividing the dynamic range into a set of discrete portions; and generating a linear approximation for each discrete portion of the activation function.

[0114] In some aspects, dividing the dynamic range into a set of discrete portions includes dividing the activation function into a number of discrete portions, the number selected based on an accuracy of the piecewise linear approximation and a computational cost of the piecewise linear approximation.

[0115] In some aspects, method 500 further includes quantizing the activation data to a smaller bit width.

[0116] In some aspects, method 500 also includes outputting the quantized activation data to a subsequent layer of the neural network.

[0117] In some aspects, method 500 also includes outputting activation data from a layer of the neural network.

[0118] It should be noted that Figure 5 This is merely one example of a method, and other methods including fewer, additional, or alternative steps may also be consistent with the present disclosure.

[0119] Example Processing System for Quantized Machine Learning

[0120] In some respects, reference Figures 1 to 5 The described workflows, techniques, and methods may be implemented on one or more devices or systems. Figure 6 An example processing system 600 is depicted that is configured to perform various aspects of the present disclosure, including, for example, with respect to Figures 1 to 5 In one aspect, the processing system 600 may correspond to a system such as a processor that uses Figure 1 The quantization architecture 100 and / or a piecewise linear approximation of the activation function (such as Figure 2A Piecewise linear approximation 205 and Figure 2B A machine learning system that trains and / or uses a quantized machine learning model using a piecewise linear approximation 250 of a machine learning model. In at least some aspects, as discussed above, the operations described below with respect to the processing system 600 may be distributed across any number of devices.

[0121] The processing system 600 includes a central processing unit (CPU) 602, which may be a multi-core CPU in some examples. Instructions executed at the CPU 602 may be loaded, for example, from a program memory associated with the CPU 602, or may be loaded from a memory partition (e.g., in the memory 624).

[0122] The processing system 600 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 604 , a digital signal processor (DSP) 606 , a neural processing unit (NPU) 608 , a multimedia processing unit 610 , and a wireless connectivity component 612 .

[0123] An NPU, such as NPU 608, is generally a specialized circuit configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0124] An NPU, such as NPU 608, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, the NPU may be part of a dedicated neural network accelerator.

[0125] The NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPUs that can perform both training and inference, the two tasks can generally still be performed independently.

[0126] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters (such as weights and biases) to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0127] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs can be configured to input a new data segment and quickly process this segment through an already trained model to generate a model output (e.g., inference).

[0128] In one specific implementation, the NPU 608 is part of one or more of the CPU 602 , the GPU 604 , and / or the DSP 606 .

[0129] In some examples, wireless connectivity component 612 may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity component 612 is further connected to one or more antennas 614.

[0130] The processing system 600 may also include one or more sensor processing units 616 associated with any manner of sensors, one or more image signal processors (ISPs) 618 associated with any manner of image sensors, and / or a navigation processor 620, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0131] The processing system 600 may also include one or more input and / or output devices 622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0132] In some examples, one or more of the processors of processing system 600 may be based on the ARM or RISC-V instruction set.

[0133] The processing system 600 also includes a memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 624 includes computer-executable components that are executable by one or more of the aforementioned processors of the processing system 600.

[0134] Specifically, in this example, the memory 624 includes a multiplication component 624A, a dequantization component 624B, an activation component 624C, and a quantization component 624D. The memory 624 also includes a set of quantization parameters 624E and model parameters 624F. The model parameters 624F may generally correspond to learned parameters of all or a portion of a machine learning model, such as weights and / or biases for one or more layers of a neural network. The quantization parameters 624E generally correspond to statistics or parameters for controlling quantization operations, such as scaling factors and / or zero points for each tensor (e.g., for input tensors, output activations, weights and / or biases for one or more layers of the model). As discussed above, the quantization parameters 624E and model parameters 624F may be learned during training of the model. The depicted components, as well as other components not depicted, may be configured to perform various aspects of the techniques described herein. Although not described herein for conceptual clarity, the quantization parameters 624E and model parameters 624F may be described herein for various aspects of the techniques described herein. Figure 6 Although depicted as discrete components in the drawings, these components may be implemented jointly or separately in various aspects.

[0135] Processing system 600 also includes multiplication circuit 626, inverse quantization circuit 627, activation circuit 628, and quantization circuit 629. The depicted circuits, as well as other circuits not depicted, may be configured to perform various aspects of the techniques described herein.

[0136] For example, multiplication component 624A and multiplication circuit 626 may be used to perform integer and / or matrix multiplication (e.g., using accumulators and / or Figure 1 In some aspects, as discussed above, multiplication component 624A and multiplication circuit 626 can use mantissa-based operations that can reduce or eliminate the need for floating-point operations. Dequantization component 624B and dequantization circuit 627 can be used to dequantize the value (e.g., based on quantization parameter 624E). Activation component 624C and activation circuit 628 can be used to apply an activation function (such as Figure 1 activation function 124) and / or approximate activation functions (such as Figure 2A Piecewise linear approximation 205 and Figure 2B In some aspects, activation component 624C and activation circuit 628 may additionally or alternatively be used to generate a piecewise linear approximation, as discussed above. Quantization parameter 624E and quantization circuit 629 may be used to quantize values ​​(e.g., to generate Figure 1 The quantized output 140) is provided for subsequent processing.

[0137] Although for the sake of clarity Figure 6 Although depicted as separate components and circuits, the multiplication circuit 626, the inverse quantization circuit 627, the activation circuit 628, and the quantization circuit 629 may be implemented collectively or individually in other processing devices of the processing system 600, such as within the CPU 602, the GPU 604, the DSP 606, the NPU 608, etc.

[0138] Generally, the processing system 600 and / or its components may be configured to perform the methods described herein.

[0139] It is worth noting that in other aspects, aspects of the processing system 600 may be omitted, such as where the processing system 600 is a server computer, etc. For example, in other aspects, the multimedia processing unit 610, the wireless connectivity component 612, the sensor processing unit 616, the ISP 618, and / or the navigation processor 620 may be omitted. Furthermore, aspects of the processing system 600 may be distributed among multiple devices.

[0140] Sample Clauses

[0141] Clause 1: A method comprising: accessing a quantized input matrix at a layer of a neural network; generating a first intermediate value in an accumulator by performing matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator; normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; dequantizing the normalized first intermediate value; generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; and generating activation data by applying an activation function to the second intermediate value.

[0142] Clause 2: The method of clause 1, wherein normalizing the first intermediate value comprises: determining a number of leading sign bits of the first intermediate value; and left shifting the first intermediate value by the determined number of leading sign bits.

[0143] Clause 3: A method as recited in any one of Clauses 1 to 2, wherein the second intermediate value has a smaller bit width than the normalized first intermediate value.

[0144] Clause 4: The method of any of clauses 1 to 3, wherein the bit width of the normalized first intermediate value is at least twice the bit width of the second intermediate value.

[0145] Clause 5: A method according to any one of clauses 1 to 4, wherein: the normalized first intermediate value has a bit width of 32 bits; and the second intermediate value has a bit width of 16 bits.

[0146] Clause 6: A method according to any of clauses 1 to 5, wherein applying the activation function comprises applying a piecewise linear approximation of the activation function.

[0147] Clause 7: A method according to any one of clauses 1 to 6, wherein the piecewise linear approximation of the activation function is generated by: identifying a dynamic range of the activation function, outside which values ​​of the activation function are within a threshold distance from one or more fixed values; dividing the dynamic range into a set of discrete portions; and generating a linear approximation of each discrete portion of the activation function.

[0148] Clause 8: A method according to any one of clauses 1 to 7, wherein dividing the dynamic range into the set of discrete parts includes dividing the activation function into a certain number of discrete parts, the number being selected based on the accuracy of the piecewise linear approximation and the computational cost of the piecewise linear approximation.

[0149] Clause 9: A method according to any one of clauses 1 to 8, further comprising: quantizing the activation data to a smaller bit width.

[0150] Clause 10: A method according to any one of clauses 1 to 9, further comprising: outputting the quantized activation data to a subsequent layer of the neural network.

[0151] Clause 11: A method according to any one of clauses 1 to 10, further comprising: outputting the activation data from the layer of the neural network.

[0152] Clause 12: A system comprising: a memory comprising computer executable instructions; and one or more processors configured to execute the computer executable instructions and cause the processing system to perform the method according to any one of clauses 1 to 11.

[0153] Clause 13: A system comprising: means for performing the method according to any one of clauses 1 to 11.

[0154] Clause 14: A non-transitory computer readable medium comprising: computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 11.

[0155] Clause 15: A computer program product embodied on a computer readable storage medium, the computer program product comprising code for performing the method according to any one of clauses 1 to 11.

[0156] Additional considerations

[0157] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace or add various procedures or components as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality or structures and functionality that are supplementary or alternative to the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the present invention.

[0158] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0159] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). For example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0160] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Additionally, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0161] As used herein, in the context of sharing electronic signals and data between the various elements described herein, the term "connected to" can generally mean data communication between the corresponding elements connected to each other. In some cases, the elements can be directly connected to each other, such as via one or more conductive traces, lines or other conductive carriers that can carry signals and / or data between the corresponding elements directly connected to each other. In other cases, the elements can be indirectly connected to each other, such as via one or more data buses or similar shared circuits and / or integrated circuit elements for communicating signals and data between the corresponding elements indirectly connected to each other.

[0162] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0163] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, the reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "part for..." is used to explicitly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A computer-implemented method, the method comprising: Access quantized input matrices at the layers of a neural network; generating a first intermediate value in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator; normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; Dequantizing the normalized first intermediate value; generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; as well as Activation data is generated by applying an activation function to the second intermediate value.

2. The computer-implemented method of claim 1 , wherein normalizing the first intermediate value comprises: determining a number of leading sign bits of the first intermediate value; as well as The first intermediate value is left shifted by the determined number of leading sign bits. 3 . The computer-implemented method of claim 1 , wherein the second intermediate value has a smaller bit width than the normalized first intermediate value. 4 . The computer-implemented method of claim 3 , wherein a bit width of the normalized first intermediate value is at least twice a bit width of the second intermediate value.

5. The computer-implemented method of claim 3, wherein: The normalized first intermediate value has a bit width of 32 bits; and The second intermediate value has a bit width of 16 bits.

6. The computer-implemented method of claim 1, wherein applying the activation function comprises applying a piecewise linear approximation of the activation function.

7. The computer-implemented method of claim 6, wherein the piecewise linear approximation of the activation function is generated by: identifying a dynamic range of the activation function, wherein outside the dynamic range, values ​​of the activation function are within a threshold distance from one or more fixed values; dividing the dynamic range into a set of discrete portions; and Generate a linear approximation of each discrete part of the activation function.

8. The computer-implemented method of claim 7 , wherein dividing the dynamic range into the set of discrete portions comprises dividing the activation function into a number of discrete portions, the number being selected based on an accuracy of the piecewise linear approximation and a computational cost of the piecewise linear approximation.

9. The computer-implemented method of claim 1 , further comprising: The activation data is quantized to a smaller bit width.

10. The computer-implemented method of claim 9, further comprising: The quantized activation data is output to a subsequent layer of the neural network.

11. The computer-implemented method of claim 1 , further comprising: The activation data is output from the layer of the neural network.

12. A processing system, the processing system comprising: a memory including computer executable instructions; as well as One or more processors configured to execute the computer-executable instructions and cause the processing system to perform operations including: Access quantized input matrices at the layers of a neural network; generating a first intermediate value in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator; normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; Dequantizing the normalized first intermediate value; generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; as well as Activation data is generated by applying an activation function to the second intermediate value.

13. The processing system of claim 12, wherein normalizing the first intermediate value comprises: determining a number of leading sign bits of the first intermediate value; as well as The first intermediate value is left shifted by the determined number of leading sign bits.

14. The processing system of claim 12, wherein a bit width of the normalized first intermediate value is at least twice a bit width of the second intermediate value.

15. The processing system of claim 12, wherein applying the activation function comprises applying a piecewise linear approximation of the activation function.

16. The processing system of claim 15, wherein the piecewise linear approximation of the activation function is generated by: identifying a dynamic range of the activation function, wherein outside the dynamic range, values ​​of the activation function are within a threshold distance from one or more fixed values; dividing the dynamic range into a set of discrete portions; and Generate a linear approximation of each discrete part of the activation function.

17. The processing system of claim 16, wherein dividing the dynamic range into the set of discrete portions comprises dividing the activation function into a number of discrete portions, the number being selected based on an accuracy of the piecewise linear approximation and a computational cost of the piecewise linear approximation.

18. The processing system of claim 12, the operations further comprising: The activation data is quantized to a smaller bit width.

19. The processing system of claim 18, the operations further comprising: The quantized activation data is output to a subsequent layer of the neural network.

20. The processing system of claim 12, the operations further comprising: The activation data is output from the layer of the neural network.

21. A non-transitory computer readable medium comprising computer executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations comprising: Access quantized input matrices at the layers of a neural network; generating a first intermediate value in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator; normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; Dequantizing the normalized first intermediate value; generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; as well as Activation data is generated by applying an activation function to the second intermediate value.

22. The non-transitory computer readable medium of claim 21 , wherein normalizing the first intermediate value comprises: determining a number of leading sign bits of the first intermediate value; as well as The first intermediate value is left shifted by the determined number of leading sign bits.

23. The non-transitory computer readable medium of claim 21, wherein a bit width of the normalized first intermediate value is at least twice a bit width of the second intermediate value.

24. The non-transitory computer-readable medium of claim 21, wherein applying the activation function comprises applying a piecewise linear approximation of the activation function.

25. The non-transitory computer readable medium of claim 24, wherein the piecewise linear approximation of the activation function is generated by: identifying a dynamic range of the activation function, wherein outside the dynamic range, values ​​of the activation function are within a threshold distance from one or more fixed values; dividing the dynamic range into a set of discrete portions; and Generate a linear approximation of each discrete part of the activation function.

26. The non-transitory computer-readable medium of claim 25, wherein dividing the dynamic range into the set of discrete portions comprises dividing the activation function into a number of discrete portions, the number being selected based on an accuracy of the piecewise linear approximation and a computational cost of the piecewise linear approximation.

27. The non-transitory computer readable medium of claim 21, the operations further comprising: The activation data is quantized to a smaller bit width.

28. The non-transitory computer readable medium of claim 27, the operations further comprising: The quantized activation data is output to a subsequent layer of the neural network.

29. The non-transitory computer readable medium of claim 21, the operations further comprising: The activation data is output from the layer of the neural network.

30. A processing system, the processing system comprising: Means for accessing quantized input matrices at a layer of a neural network; means for generating a first intermediate value in an accumulator by performing a matrix multiplication of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network using the accumulator; means for normalizing the first intermediate value based at least in part on one or more leading sign bits of the first intermediate value; means for dequantizing the normalized first intermediate value; means for generating a second intermediate value by applying a round right shift operation to the dequantized normalized first intermediate value; as well as means for generating activation data by applying an activation function to said second intermediate value.

Images