Improved quantization for neural networks

Abstract

The invention relates to improved quantization for neural networks. Systems and methods for performing post-training quantization are disclosed herein. A processor (1002) obtains a fixed point output value from a layer of an artificial neural network, ANN, (101B), wherein the layer includes a fixed point weight determined based on a floating point weight and a weight scaling factor determined based on an output scaling factor. Next, the processor (1002) converts the fixed point output value to a floating point output value based on the output scaling factor. The processor (1002) then extends a range of floating point values. Next, the processor (1002) calculates a new output scaling factor based on the extended range of floating point output values. Finally, the processor (1002) stores the new output scaling factor in an associated memory (1003).

Description

[0001] Related Applications

[0002] This application relates to and claims the priority benefit of Indian Provisional Patent Application No. 202341052481 filed on August 4, 2023 and entitled “Methods to Improve Accuracy of Neural Networks with FixedPoint Hardware”, which is incorporated herein by reference in its entirety. Technical Field

[0003] Aspects of the present disclosure relate to the field of computing hardware and software, and more particularly to quantizing floating point values ​​for neural networks. Background Art

[0004] Quantization describes the process of limiting continuous or other large value groups to discrete groups. For example, the 32-bit weight values ​​used by the nodes of a floating-point deep neural network (DNN) can be quantized to 8 bits of precision. Advantageously, quantizing the floating-point weight values ​​of a DNN to fixed-point numbers allows the operation of the DNN to be accelerated via a fixed-point hardware accelerator.

[0005] Traditionally, floating-point DNNs are asymmetric quantized. Asymmetric quantization describes a nonlinear mapping process that distributes the weight values ​​of a DNN unevenly across a determined zero point. In the context of machine learning, asymmetric quantization is the preferred quantization method because it allows for better representation of values ​​associated with the DNN.

[0006] Unfortunately, conventional methods for quantizing floating-point DNNs are prone to various errors, such as quantization error and clipping error. Quantization error describes the difference between a floating-point value and its fixed-point representation. The goal of quantization is to reduce the quantization error so that the difference between a floating-point value and its fixed-point representation is minimized. Therefore, considering the quantization error may lead to clipping of extreme values ​​(i.e., clipping error). Therefore, quantization methods that introduce quantization error and, in turn, clipping error lead to a degradation in the accuracy of the DNN. Summary of the invention

[0007] Disclosed herein are techniques for improving the accuracy of neural networks employed on fixed-point hardware. Various embodiments include a computer-implemented method for performing post-training quantization. A processing circuit system of a suitable computing system obtains fixed-point output values ​​from a layer of an artificial neural network (ANN), wherein the layer comprises fixed-point weights and weight scaling factors. The fixed-point weights are determined based on floating-point weights, and the weight scaling factors are determined based on output scaling factors. Next, the processing circuit system converts the fixed-point output values ​​to floating-point output values ​​based on the output scaling factors. Subsequently, the processing circuit system extends the range of the floating-point output values. Next, the processing circuit system calculates a new output scaling factor based on the extended range of the floating-point output values. Finally, the processing circuit system stores the new output scaling factor in a memory associated with the suitable computing system.

[0008] This summary is provided to introduce in a simplified form a series of concepts that are further described in the technical disclosure below. It is understood that this summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, but emphasis is placed on clearly illustrating the principles of the present disclosure. In addition, in the drawings, similar reference numerals indicate corresponding parts throughout several views. Although several embodiments are described in conjunction with these drawings, the present disclosure is not limited to the embodiments disclosed herein. On the contrary, all alternatives, modifications and equivalents are intended to be covered.

[0010] Figure 1 The quantization architecture in an implementation scheme is shown.

[0011] Figure 2 A quantification method in an embodiment is shown.

[0012] Figure 3 Another quantization architecture in an embodiment is shown.

[0013] Figure 4 The operation sequence in the embodiment is shown.

[0014] Figure 5 The post-training quantization process in an implementation scheme is shown.

[0015] Figure 6 The scaling process in an implementation scheme is shown.

[0016] Figure 7 The saturation process in an embodiment is shown.

[0017] Figure 8A result table in an embodiment is shown.

[0018] Fig. 9 Another results table in an embodiment is shown.

[0019] Fig.10 A computing system suitable for implementing the various operating environments, architectures, processes, scenarios, and sequences discussed below with respect to other figures is shown. DETAILED DESCRIPTION

[0020] Disclosed herein is a technique for providing a quantization method that can potentially improve the accuracy of neural networks employed via fixed-point hardware. Generally, deep neural networks (DNNs) are trained to perform specified tasks using floating-point values. For example, such tasks may include object detection, speech recognition, image processing, language modeling, etc. Therefore, the weight values ​​employed by the nodes of the trained DNN represent floating-point numbers.

[0021] In some implementations, the floating point weights of the DNN are quantized to fixed point values ​​via a process called post-training quantization (PTQ). The goal of the PTQ process is to quantize the weight values ​​of the DNN to the desired bit precision while maintaining the accuracy of the neural network. The purpose of the PTQ process is to allow the operation of the DNN to be accelerated via a fixed point hardware accelerator.

[0022] Disclosed herein is a method for performing post-training quantization that can potentially improve the accuracy, scaling, latency, bandwidth, and power characteristics of a DNN. In an embodiment, the disclosed PTQ process represents an iterative loop of two subprocesses. The first subprocess represents a scaling process that describes a method in which the floating-point weight values ​​of a DNN are quantized to fixed-point numbers. The second subprocess represents a saturation process that describes a method in which various errors are taken into account. In an embodiment, a processor iteratively cycles through the scaling process and the saturation process until a number of calibration iterations have been reached. Once reached, the processor embeds the fixed-point weights determined via the scaling process and the saturation process into the DNN.

[0023] Now turning to the pictures, Figure 1 Quantization architecture 100 in an embodiment is shown. Quantization architecture 100 represents an exemplary software architecture for quantizing a floating point neural network. Quantization architecture 100 can be implemented in the context of program instructions that can be executed by a suitable computing system. For example, such a computing system can include a microcontroller, a digital signal processor, an application specific integrated circuit, a central processing unit, a graphics processing unit, a field programmable gate array, and / or any other processing resource. Quantization architecture 100 includes, but is not limited to, a floating point inference model 101A, a fixed point inference model 101B, and a post-training quantization (PTQ) process 103.

[0024] Floating point inference model 101A represents a neural network that performs operations using floating point weights. For example, floating point inference model 101A may represent a convolutional neural network (CNN), an artificial neural network (ANN), or another similar deep neural network (DNN). In an embodiment, floating point inference model 101A is iteratively trained using floating point data to perform tasks. For example, such tasks may include classification, sequence learning, or function approximation.

[0025] Fixed-point inference model 101B is a fixed-point representation of floating-point inference model 101 A. Thus, fixed-point inference model 101B represents a neural network that performs the same operations as floating-point inference model 101A but instead utilizes fixed-point weights. In an embodiment, fixed-point inference model 101B offloads fixed-point computations to a hardware accelerator that is configured to perform the operations of fixed-point inference model 101B.

[0026] Post-training quantization process 103 represents software that quantizes the weight values ​​of the neural network while maintaining the accuracy of the network. For example, PTQ process 103 may be executed to convert the floating-point weights of floating-point inference model 101A to fixed-point values. In another example, PTQ process 103 may be executed to improve the accuracy of fixed-point inference model 101B. In an embodiment, PTQ process 103 represents software that, when executed by suitable processing circuitry, directs processing circuitry to perform a quantization of the weight values ​​of the neural network based on a later inference inference model 101A. Figure 2 The program instructions for executing the steps of the PTQ process 103 discussed in.

[0027] In a brief operational scenario, the PTQ process 103 may be executed to convert the floating point reasoning model 101A to the fixed point reasoning model 101B. First, the floating point reasoning model 101A receives floating point input data representing image data, video data, audio data, language / text data, or a combination thereof. In response, the floating point reasoning model 101A generates floating point output data, which is supplied to the PTQ process 103. The PTQ process 103 utilizes the floating point data (input, output, and weights) of the floating point reasoning model 101A to generate a first set of fixed point weights.

[0028] Next, the PTQ process 103 loads the first set of fixed-point weights to the fixed-point inference model 101B. Once accepted by the layers of the network, the fixed-point inference model 101B receives the fixed-point input data and generates a first set of fixed-point output data. The fixed-point inference model 101B supplies the first set of fixed-point output data to the PTQ process 103. The PTQ process 103 analyzes the accuracy of the first set of fixed-point output data, and in response, adjusts the first set of fixed-point weights to generate a second set of fixed-point weights. The PTQ process 103 loads the second set of fixed-point weights to the fixed-point inference model 101B, which generates the second set of fixed-point output data. In an embodiment, the PTQ process 103 continues to adjust subsequent sets of fixed-point weights until the output data of the fixed-point inference model 101B meets the accuracy threshold.

[0029] Figure 2 Quantization method 200 in an embodiment is shown. Quantization method 200 represents software for quantizing the weight values ​​of a neural network while maintaining the accuracy of the network. For example, quantization method 200 may represent Figure 1 The PTQ process 103 of the embodiment of the present invention. The quantization method 200 may be implemented in the context of program instructions that, when executed by a suitable computing system, direct the processing circuitry of the computing system to operate as follows, with reference to Figure 2 In some examples, the quantization method 200 may include additional operations and / or less than Figure 2 All the operations shown in .

[0030] First, the processing circuit system inputs a floating point input value to a floating point inference model configured with floating point weights (step 201). For example, the processing circuit system may supply the floating point input value to the floating point inference model 101A. The floating point input value supplied to the floating point inference model represents data collected by a source. For example, the input data may represent image data, video data, or audio data collected by a corresponding sensor, or language / text data received as input. The floating point inference model receives the floating point input data and, in response, generates floating point output data to be supplied to the processing circuit system. In an embodiment, the floating point inference model provides the floating point input data, output data, and weight data to the processing circuit system.

[0031] Next, the processing circuit system calculates an input scaling factor, an output scaling factor, and a weight scaling factor based on the received floating point data (step 203). The input scaling factor, the output scaling factor, and the weight scaling factor represent scales for quantizing the corresponding floating point data. For example, the input scaling factor can be used to quantize the floating point input data, the output scaling factor can be used to quantize the floating point output data, and the weight scaling factor can be used to quantize the floating point weight.

[0032] In an embodiment, in order to determine the input scaling factor, the output scaling factor and the weight scaling factor, the processing circuit system first determines the total number of fixed-point values ​​that can be used to represent the fixed-point data. For example, if the associated hardware accelerator performs operations using signed 8-bit data (i.e., -128 to 127), the total number of values ​​that are allowed to represent the fixed-point data (e.g., input data, output data, or weight data) is equal to 256. Next, in order to determine the input scaling factor, the processing circuit system divides the allowable number of fixed-point values ​​by the range of floating-point input values. The range of floating-point input values ​​describes the difference between the maximum floating-point input value and the minimum floating-point input value. Similarly, in order to determine the output scaling factor, the processing circuit system divides the allowable number of fixed-point values ​​by the range of floating-point output values. The range of floating-point output values ​​describes the difference between the maximum floating-point output value and the minimum floating-point output value. Additionally or alternatively, in order to calculate the weight scaling factor, the processing circuit system divides the allowable number of fixed-point values ​​by the absolute maximum floating-point weight value.

[0033] After the processing circuit system calculates the scaling factor for the floating-point data, the processing circuit system obtains a fixed-point output value based on the current output scaling factor (step 205). For example, the processing circuit system may scale the received floating-point output data using the current output scaling factor to obtain a fixed-point output value. In another example, to obtain the fixed-point output value, the processing circuit system first scales the floating-point weight to a fixed-point value via the current weight scaling factor. Next, the processing circuit system supplies the fixed-point weight to the fixed-point inference model. For example, the processing circuit system may supply the fixed-point weight to the fixed-point inference model 101B. Once the fixed-point weight is accepted by the layer of the fixed-point inference model, the processing circuit system inputs the fixed-point input value to the fixed-point inference model. In an embodiment, the fixed-point input value is generated by the processing circuit system by scaling the floating-point input value using the input scaling factor. In response to receiving the fixed-point input value, the fixed-point inference model generates a fixed-point output value.

[0034] Next, the processing circuit system determines whether the saturation level of the fixed-point output value represents acceptable saturation (step 207). The saturation level of the fixed-point output value describes the number of fixed-point output values ​​represented by the extreme values ​​of the allowable fixed-point range. For example, if the data type employed represents signed 8-bit data, the saturation level describes the number of fixed-point output values ​​represented by -128 or 127. In another example, when the data type employed represents unsigned 8-bit data, the saturation level describes the number of fixed-point output values ​​represented by 0 or 255.

[0035] In an embodiment, if the processing circuitry determines that less than (or equal to) a threshold level / number / percentage (e.g., 1%) of fixed-point output values ​​are saturated, then the processing circuitry accepts the current fixed-point weights (step 208). The processing circuitry may accept the current fixed-point weights by embedding the weights into a layer of the fixed-point inference model.

[0036] Alternatively, if the processing circuitry determines that more than a threshold level / number / percentage (e.g., 1%) of fixed-point output values ​​are saturated, then the processing circuitry converts the fixed-point output values ​​to floating-point output values ​​based on the current output scaling factor (step 209). Thus, the floating-point output values ​​produced by the processing circuitry represent a second set of floating-point output values.

[0037] After generating the second set of floating-point output values, the processing circuitry expands the range of the second set based on the direction of saturation (step 211). For example, if the fixed-point output values ​​saturate at a high end of the allowable fixed-point range (e.g., 127 or 255), then the processing circuitry increases the maximum value represented by the second set of floating-point output values ​​by a threshold amount. In an embodiment, the processing circuitry increases the maximum floating-point output value by a factor of 2%. Alternatively, if the fixed-point output values ​​saturate at a low end of the allowable fixed-point range (e.g., -128 or 0), then the processing circuitry decreases the minimum value represented by the second set of floating-point output values ​​by a threshold amount (e.g., 2%).

[0038] Next, the processing circuitry calculates a new output scaling factor based on the extended range of the second set of floating-point output values ​​(step 213). In an embodiment, to calculate the new output scaling factor, the processing circuitry divides the allowable number of fixed-point values ​​by the extended range of the second set of floating-point output values. After calculating the new output scaling factor, the processing circuitry stores the new output scaling factor in an associated memory (step 215).

[0039] Next, the processing circuit system calculates a new weight scaling factor based on the new output scaling factor (step 217). The processing circuit system may store the new weight scaling factor to a memory. Once the new weight scaling factor is calculated, the processing circuit system updates the fixed-point weights based on the new weight scaling factor (step 219), and returns to step 205. In an embodiment, the processing circuit system iteratively loops through the quantization process 200 until the floating-point output data indicates acceptable saturation. In another embodiment, the processing circuit system iteratively loops through the quantization process 200 until a maximum number of calibration iterations has been reached.

[0040] Figure 3 Quantization architecture 300 in an embodiment is shown. Quantization architecture 300 represents an exemplary software architecture for quantizing floating point data of a neural network while maintaining the accuracy of the network. For example, quantization architecture 300 may represent Figure 1 Quantization architecture 100. Quantization architecture 300 may be implemented in the context of program instructions executable by a suitable computing system. For example, such computing system may include a microcontroller, a digital signal processor, an application specific integrated circuit, a central processing unit, a graphics processing unit, a field programmable gate array, and / or any other processing resource. Quantization architecture 300 includes, but is not limited to, an inference model 301, a quantization logic 303, a scaling logic 305, and a saturation logic 307.

[0041] Inference model 301 represents a neural network that has been trained to perform a task. For example, inference model 301 may represent an ANN, a CNN, or another similar DNN that is trained to perform tasks such as object detection, image classification, speech recognition, etc. In an embodiment, inference model 301 is trained using floating point data. Therefore, the weight values ​​used by the layers of inference model 301 represent floating point values. In an embodiment, the floating point weights of inference model 301 are quantized via quantization logic 303, scaling logic 305, and saturation logic 307 to allow the operation of inference model 301 to be performed by a hardware accelerator. Advantageously, outsourcing the operation of inference model 301 to an associated hardware accelerator reduces the processing time required to perform the task.

[0042] Quantization logic 303 represents a software block that performs the post-training quantization method disclosed herein. For example, quantization logic 303 can represent PTQ process 103 or quantization method 200. Quantization logic 303 receives data from inference model 301 to generate fixed-point weights for the layers of inference model 301. In an embodiment, quantization logic 303 interfaces with scaling logic 305 and saturation logic 307 to generate fixed-point weights that maintain the accuracy of inference model 301.

[0043] Scaling logic 305 represents a software block that determines scaling factors for floating point data used to quantize inference model 301. For example, scaling logic 305 can generate scaling factors for floating point input values, floating point output values, and floating point weights of inference model 301. Scaling logic 305 receives a floating point input range, output range, and weight range from quantization logic 303. In response, scaling logic 305 calculates corresponding input scaling factors, output scaling factors, and weight scaling factors for the floating point data of inference model 301.

[0044] Saturation logic 307 represents a software block that monitors the saturation level of fixed-point output values ​​generated by inference model 301. The saturation level of a fixed-point output value describes the number of values ​​represented by the low or high end of the allowable fixed-point range. For example, if the associated hardware accelerator performs calculations using signed 8-bit data, the saturation level describes the amount of fixed-point output values ​​represented by -127 or +128. In an embodiment, if the saturation level of the fixed-point output value does not represent acceptable saturation, the saturation logic 307 extends the range of floating-point output values, and the scaling logic 305 generates a new scaling factor based on the extended range. Alternatively, if the saturation level of the fixed-point output value represents acceptable saturation, the quantization logic 303 embeds the fixed-point weights (associated with acceptable saturation) into the inference model 301.

[0045] Figure 4 4 shows an operation sequence 400 in an embodiment. The operation sequence 400 represents the operation sequence performed by the quantization architecture 300. First, the inference model 301 receives a floating point input value (X) representing image data, video data, audio data, language / text data, or a combination thereof. float Next, the inference model 301 generates a first set of floating-point output values ​​(Y float1 ). Inference model 301 provides the first set of floating point output values ​​to quantization logic 303. In an embodiment, inference model 301 also provides floating point input values ​​and floating point weights used to generate the first set of floating point output values ​​to quantization logic 303. In another embodiment, quantization logic 303 is preloaded with floating point input values ​​and floating point weights.

[0046] Next, the quantization logic 303 determines the floating point input range (X RANGEfloat ), the first floating point output range (Y RANGEfloat1 ) and the absolute maximum floating point weight value (|W float |). In an embodiment, to determine the floating point input range and the floating point output range, the quantization logic 303 determines the difference between the maximum floating point value and the minimum floating point value. For example, the quantization logic 303 may use the following equation:

[0047] (1)X RANGEfloat =X MAXfloat -X MINfloat

[0048] (2)Y RANGEfloat1 =Y MAXftoat1 -Y MINfloat1

[0049] So that in equation (1), X RANGEfloat Indicates the floating-point input range, x MAXfloat Represents the maximum floating-point input value, and XMINfloat represents the minimum floating-point input value and makes Y RANGEfloat1 Indicates the first floating point output range, Y MAXfloat1 represents the maximum value from the first set of floating-point output values, and Y MINfloat1 Represents the minimum value from the first set of floating-point output values.

[0050] After determining the floating point input range, the first floating point output range, and the absolute maximum floating point weight value, the quantization logic 303 provides the floating point range and the absolute maximum floating point weight value to the scaling logic 305. In response, the scaling logic 305 calculates the scaling factor for the floating point input value, the first set of floating point output values, and the floating point weight. In an embodiment, to calculate the scaling factor, the scaling logic 305 first determines the allowable number of fixed point values ​​that can be used to represent the fixed point data. For example, if the associated hardware accelerator performs calculations using 8-bit data, the total number of values ​​that can be used to represent the fixed point data is equal to 256. Alternatively, if the associated hardware accelerator performs calculations using 16-bit data, the total number of values ​​that can be used to represent the fixed point data is equal to 65,536.

[0051] Next, to determine the input scaling factor, scaling logic 305 divides the allowable number of fixed-point values ​​by the floating-point input range. For example, scaling logic 305 may execute the following equation:

[0052]

[0053] So that in equation (3), s X represents the input scaling factor, and X RANGEfloat Indicates the floating-point input range.

[0054] Similarly, to determine the output scaling factor, scaling logic 305 divides the allowable number of fixed-point values ​​by the floating-point output range. For example, scaling logic 305 may execute the following equation:

[0055]

[0056] So that in equation (4), s Y1 represents the first output scaling factor, and Y RANGEfloat1 Indicates the first floating-point output range.

[0057] Additionally or alternatively, to determine the weight scaling factor, the scaling logic 305 divides the allowable number of fixed-point values ​​by the absolute maximum floating-point weight value. For example, the scaling logic 305 may execute the following equation:

[0058]

[0059] So that in equation (5), sW1 represents the first weight scaling factor, and |W float | represents the absolute maximum floating point weight value.

[0060] Scaling logic 305 takes the input scaling factor (s X ), the first output scaling factor (s Y1 ) and the first weight scaling factor (s W1 ) is provided to the quantization logic 303. In response, the quantization logic 303 quantizes the floating point weights into a first set of fixed point weights (W) using the weight scaling factor. fixed1 ). Quantization logic 303 provides the first set of fixed-point weights to inference model 301. In an embodiment, quantization logic 303 also calculates the fixed-point input value (X fixed ) is provided to the inference model 301. For example, the quantization logic 303 may utilize the input scaling factor to quantize the floating point input value to an appropriate fixed point representation. Alternatively, the inference model 301 may receive the fixed point input value from an external source.

[0061] After receiving the necessary fixed-point data, the inference model 301 generates a first set of fixed-point output values ​​(Y fixed1 ), and provides the value to the quantization logic 303. In response, the quantization logic 303 supplies the first set of fixed point output values ​​and the first output scaling factor to the saturation logic 307.

[0062] Next, saturation logic 307 determines whether the first set of fixed-point output values ​​meets the saturation threshold. In an embodiment, saturation logic 307 checks the number of values ​​represented by the extremes of the acceptable fixed-point range to determine whether the first set of fixed-point output values ​​meets the saturation threshold. For example, if the associated hardware accelerator performs operations using signed 8-bit data, then saturation logic 307 checks the number of values ​​represented by -127 or +128 from the first set of fixed-point output values. In an embodiment, if more than 1% of the values ​​from the first set of fixed-point output values ​​are represented by the extremes of the acceptable fixed-point range, then saturation logic 307 determines that the first set of fixed-point values ​​does not meet the saturation threshold. Otherwise, saturation logic 307 determines that the first set of fixed-point values ​​meets the saturation threshold.

[0063] In response to determining that the first set of fixed-point output values ​​does not satisfy the saturation threshold, saturation logic 307 converts the first set of fixed-point output values ​​to a second set of floating-point output values ​​(Y float2). Next, saturation logic 307 extends the second set of floating point output values ​​in the saturated direction. For example, when the desired data type for performing fixed point calculations is signed 8-bit data, and values ​​greater than 1% from the first set of fixed point output values ​​are represented by the maximum value of the allowable fixed point range (i.e., 128), saturation logic 307 extends the maximum value represented by the second set of floating point output values ​​by a factor of 2%. Alternatively, if values ​​greater than 1% from the first set of fixed point output values ​​are represented by the minimum value of the allowable fixed point range (i.e., -127), then saturation logic 307 extends the minimum value represented by the second set of floating point output values ​​by a factor of 2%. The factor 1.02 is only one example of an expansion factor. Other values ​​may be used for the expansion factor.

[0064] Saturation logic 307 converts the second set of extended floating point output values ​​(Y float2+ ) is provided to the quantization logic 303. In response, the quantization logic 303 calculates the range of the second set of extended floating point output values ​​(Y RANGEfloat2+ ), and provides the range to the scaling logic 305. The scaling logic 305 receives the second set of extended floating point output value ranges, and responsively determines a second output scaling factor (s Y2 ) and the second weight scaling factor (s W2 ). The scaling logic 305 provides the second output scaling factor and the second weight scaling factor to the quantization logic 303. In response, the quantization logic 303 quantizes the floating point weights based on the second weight scaling factor to generate a second set of fixed point weights (W fixed2 ).

[0065] Quantization logic 303 provides a second set of fixed-point weights to inference model 301. After receiving the necessary fixed-point data, inference model 301 generates a second set of fixed-point output values ​​(Y fixed2 ), and provides the value to quantization logic 303. In response, quantization logic 303 supplies a second set of fixed-point output values ​​and a second output scaling factor to saturation logic 307. Next, saturation logic 307 determines that the second set of fixed-point output values ​​meets the saturation threshold, and responsively informs the determination to quantization logic 303. Finally, quantization logic 303 embeds the second set of fixed-point weights into the layers of inference model 301.

[0066] Now, moving on to the next picture. Figure 5 500 is shown in an embodiment. PTQ process 500 represents a process for quantizing floating point weights of a neural network while maintaining the accuracy of the network. For example, PTQ process 500 may represent PTQ process 103, quantization method 200, or quantization logic 303. In an embodiment, PTQ process 500 may be implemented in the context of program instructions that, when executed by a suitable computing system, direct the processing circuitry of the computing system to operate as follows, with reference to Figure 5 Steps in .

[0067] First, the processing circuit system obtains a floating point output value from an artificial neural network (step 501). It should be noted that the processing circuit system can obtain a floating point output value from any type of neural network, but for the purpose of explanation, ANN will be discussed herein. In an embodiment, the processing circuit system further obtains a floating point input value and a floating point weight from the ANN.

[0068] Next, the processing circuit system performs a scaling process for determining a scaling factor for the floating point data used for the ANN (step 503). For example, the processing circuit system may perform a scaling process that is later referenced. Figure 6 Scaling process 600 discussed. In an embodiment, scaling process 600 determines input scaling factors, output scaling factors, and weight scaling factors for floating-point data of an ANN. The input scaling factor represents the scale used to quantize the floating-point input data to a desired fixed-point representation. For example, the desired fixed-point representation may include signed 8-bit data, unsigned 8-bit data, signed 16-bit data, or another similar representation. Similarly, the output scaling factor and the weight scaling factor represent the scale used to quantize the floating-point output value and the floating-point weight respectively. In an embodiment, scaling process 600 outputs fixed-point input values, fixed-point output values, and fixed-point weights based on the input scaling factor, the output scaling factor, and the weight scaling factor.

[0069] After performing the scaling process 600, the processing circuitry determines whether a maximum number of calibration iterations has been reached (step 505). The number of calibration iterations describes the number of times the PTQ process 500 has been performed without performing step 506. In an embodiment, the maximum number of calibration iterations is equal to 25. Therefore, at the 25th calibration iteration, the processing circuitry will determine that the maximum number of calibrations has been reached and proceed to embed the last set of fixed-point weights produced by the scaling process 600 into the ANN (step 506).

[0070] Alternatively, before the 25th calibration iteration, the processing circuit system will determine that the maximum number of calibration iterations has not been reached and continue to perform the saturation process (step 507). The saturation process performed by the processing circuit system determines whether the saturation level of the fixed-point output value generated by the ANN meets the saturation threshold. For example, the processing circuit system may perform a later reference to Figure 7 The saturation process 700 is discussed.

[0071] In an embodiment, to determine the saturation level, saturation process 700 checks the number of fixed-point output values ​​represented by the extreme values ​​of the allowable fixed-point range. For example, when the extreme values ​​of the allowable fixed-point range are equal to -127 and +128, the processing circuit system checks the number of values ​​represented by -127 and the number of values ​​represented by +128. If the number of fixed-point output values ​​represented by one of the extreme values ​​is greater than 1%, then the processing circuit system expands the range of floating-point output values ​​based on the direction of saturation and returns to step 503. Alternatively, if the number of values ​​represented by the extreme values ​​of the fixed-point range is less than or equal to 1%, then the processing circuit system proceeds to step 506.

[0072] Figure 6 A scaling process 600 in an embodiment is shown. Scaling process 600 represents a process for quantizing floating point weights of a neural network (e.g., scaling logic 305). For purposes of explanation, scaling process 600 represents a process employed by PTQ process 500 for determining input scaling factors, output scaling factors, and weight scaling factors to quantize floating point data of the ANN. It should be noted that scaling process 600 determines scaling factors for each layer of the ANN, but for purposes of explanation, scaling process 600 will be explained within the context of a single layer.

[0073] First, the processing circuit system determines the desired data type for representing fixed-point data (step 601). For example, if the associated hardware accelerator performs calculations using signed 8-bit data, then the desired data type represents signed 8-bit data. Next, the processing circuit system calculates input scaling factors and output scaling factors based on floating-point input values ​​and floating-point output values ​​of the ANN (step 603). The input scaling factors and output scaling factors represent the scales used to quantize the corresponding floating-point input values ​​and floating-point output values.

[0074] In an embodiment, to calculate the input scaling factor and the output scaling factor, the processing circuit system first determines the total number of fixed-point values ​​that are allowed to represent the fixed-point data. For example, if the associated hardware accelerator performs operations using signed 8-bit data (i.e., -128 to 127), then the total number of values ​​that are allowed to represent the fixed-point data (e.g., input data, output data, or weight data) is equal to 256. Next, the processing circuit system determines the range of floating-point input values ​​and the range of floating-point output values. For example, the processing circuit system may execute the following equation:

[0075] (1)X RANGEfloat =x MAXfloat -x MINfloat

[0076] (2)Y RANGEfloat =Y MAXfloat -Y MINfloat

[0077] So that in equation (1), X RANGEfloat Indicates the floating-point input range, x MAXfloat Represents the maximum floating-point input value, and X MINfloat represents the minimum floating-point input value and makes Y RANGEfloat Indicates the floating point output range, Y MAXfloat Indicates the maximum floating-point output value, and Y MINfloat Indicates the minimum floating-point output value.

[0078] Next, to determine the input scaling factor and the output scaling factor, the processing circuit system divides the allowable number of fixed-point values ​​by the corresponding floating-point range. For example, the processing circuit system may execute the following equation:

[0079]

[0080]

[0081] So that in equation (3), s X represents the input scaling factor, and X range represents the range of floating-point input values, and makes it so that in equation (4), s Y represents the output scaling factor, and Y range Indicates the range of floating-point output values.

[0082] In an embodiment, the processing circuit system also calculates zero points for floating-point input values ​​and floating-point output values. To determine the zero points, the processing circuit system first determines the maximum fixed-point value allowed to represent floating-point data. For example, if the desired data type represents signed 8-bit data, then the maximum fixed-point value allowed to represent floating-point data is equal to 127. Next, to determine the input zero point and the output zero point, the processing circuit system may execute the following equations:

[0083] (5) Z X = Maximum fixed point value - (X MAXfloat *s X )

[0084] (6)z Y = Maximum fixed point value - (Y MAXfloat *s Y )

[0085] So that in equation (5), z X Indicates input zero point, x MAXfloat Represents the maximum floating-point input value, and s X represents the input scaling factor and makes z Y Indicates output zero point, Y MAXfloatIndicates the maximum floating-point output value, and s Y Indicates the output scaling factor.

[0086] Next, the processing circuit system calculates a weight scaling factor for the floating point weights of the ANN (step 605). In an embodiment, to calculate the weight scaling factor, the processing circuit system divides the allowable number of fixed point values ​​by the absolute maximum floating point weight value. Therefore, the weight scaling factor can be determined using the following equation:

[0087]

[0088] So that in equation (7), s w represents the weight scaling factor, and |W min / max | represents the absolute maximum floating point weight value.

[0089] After calculating the scaling factors for the floating point input values, output values, and weights, the processing circuit system approximates the scaling ratio based on the hardware scaling ratio utilized by the associated fixed point hardware (step 607). For example, at run time, the ANN may offload various multiply and accumulate (MAC) operations to the hardware accelerator. For example, such MAC operations may include convolution operations, inner product operations, matrix multiplication operations, or other similar operations. Although the hardware accelerator performs fixed point calculations, the hardware accelerator may require the hardware scaling ratio to scale the output of the calculation back to the desired bit depth. In an embodiment, to approximate the scaling ratio, the processing circuit system executes the following equation:

[0090]

[0091] So that in equation (8), s Y represents the output scaling factor, s X represents the input scaling factor, s W Represents the weight scaling factor, HWA scale Represents the hardware scaling factor, and HWA shift represents the hardware shift factor. In addition, it should be noted that The ratio of represents the scaling ratio, while The ratio represents the hardware scaling ratio.

[0092] Next, the processing circuitry determines whether the error in the approximate scale ratio is below an error threshold (step 609). In an embodiment, to determine the amount of error between the scale ratio and the hardware scale ratio, the processing circuitry checks the difference between the scale ratio and the hardware scale ratio. If the difference between the scale ratio and the hardware scale ratio is below the error threshold amount, the processing circuitry quantizes the floating point weights using a weight scaling factor (step 611). Alternatively, if the difference between the scale ratio and the hardware scale ratio is above the error threshold amount, the processing circuitry adjusts the weight scaling factor accordingly (step 610).

[0093] In an embodiment, to adjust the weight scaling factor, the processing circuit system first calculates the ideal weight scaling factor based on the hardware scaling ratio. For example, the processing circuit system may execute the following equation:

[0094]

[0095] So that in equation (9), s W ′ represents the ideal weight scaling factor, s Y Indicates the output scaling factor, HWA shift Indicates the hardware shift factor, s X represents the input scaling factor, and HWA scale Indicates the hardware scaling factor.

[0096] Next, the processing circuit system determines whether the difference between the ideal weight scaling factor and the weight scaling factor is less than 2%. If the difference between the ideal weight scaling factor and the weight scaling factor is less than 2%, the processing circuit system replaces the weight scaling factor with the ideal weight scaling factor. Alternatively, if the difference between the ideal weight scaling factor and the weight scaling factor is greater than or equal to 2%, the processing circuit system determines whether the weight scaling factor should be increased by 2% or decreased by 2%. In an embodiment, in order to determine whether to increase or decrease the weight scaling factor, the processing circuit system calculates the difference between the ideal weight scaling factor and the weight scaling factor. If the difference between the ideal weight scaling factor and the weight scaling factor is greater than zero, the processing circuit system increases the weight scaling factor by a factor of 2%. Alternatively, if the difference between the ideal weight scaling factor and the weight scaling factor is less than or equal to zero, the processing circuit system reduces the weight scaling factor by a factor of 2%.

[0097] After adjusting the weight scaling factor, the processing circuit system quantizes the floating point weight based on the adjusted weight scaling factor (step 611). Next, the processing circuit system calculates a hardware bias term based on the weight scaling factor (step 613). The hardware bias term describes a value utilized by an associated hardware accelerator to perform a MAC operation. In an embodiment, to calculate the hardware bias term, the processing circuit system utilizes the following equation:

[0098]

[0099] So that in equation (10), B1 represents the hardware bias term, B′ represents the quantized bias term, W′ represents the fixed-point weight, and z X Indicates input zero point, z Y indicates the output zero point, and It should be noted that the quantized bias term is a known value.

[0100] Next, the processing circuit system determines whether the number of bits for representing the hardware bias term is greater than the number of bits allocated by the associated accumulator (step 615). For example, if the associated accumulator performs calculations using 32-bit data, the number of bits allowed to represent the hardware bias term is equal to 30 bits. If the number of bits for representing the hardware bias term is greater than the allocated amount, the processing circuit system reduces the number of bits representing the hardware bias term by one time (step 617). In an embodiment, the processing circuit system divides the weight scaling factor by two to reduce the number of bits for representing the hardware bias term. Advantageously, reducing the number of hardware bias bits prevents an accumulator overflow error. Next, the processing circuit system returns to step 607 to ensure that the adjusted weight scaling factor meets the hardware requirements of the associated hardware accelerator. The processing circuit system iteratively performs steps 607 to 617 until the number of bits for representing the hardware bias term is less than or equal to the allocated amount.

[0101] Once the number of bits used to represent the hardware bias term is less than or equal to the allocated amount, the processing circuit system embeds the fixed-point weights into the ANN (step 616). Next, in the context of the PTQ process 500, the processing circuit system determines whether a maximum number of calibration iterations has been reached (step 505). When the maximum number of calibrations has not been reached, the processing circuit system continues to perform the saturation process 700 (step 507).

[0102] Figure 7 The saturation process 700 in an embodiment is shown. The saturation process 700 represents a process for determining whether the saturation level of a fixed-point output value (produced by the ANN) represents acceptable saturation. For example, the saturation process 700 may represent Figure 3 Saturation logic 307. For purposes of explanation, saturation process 700 represents the process employed by PTQ process 500 for determining the saturation level of a fixed-point output value produced by the ANN.

[0103] First, the processing circuit system obtains fixed-point output values ​​from the ANN (step 701). In an embodiment, after the processing circuit system embeds the fixed-point weights into the layers of the ANN, the processing circuit system provides the fixed-point input values ​​to the ANN. For example, the processing circuit system may utilize the input scaling factor and input zero point generated by the scaling process 600 to quantize the floating-point input values ​​to fixed-point input values, which are then provided to the ANN. Alternatively, the ANN may receive fixed-point input values ​​from an external source, such as a memory configured to store fixed-point input data. In response to receiving the necessary fixed-point data, the ANN generates fixed-point output values, which are obtained by the processing circuit system.

[0104] Next, the processing circuit system determines whether the saturation level of the fixed-point output value represents acceptable saturation (step 703). The saturation level describes the number of fixed-point output values ​​represented by the extreme values ​​of the allowable fixed-point range. For example, when the extreme values ​​of the allowable fixed-point range are equal to -127 and +128, the processing circuit system checks the number of values ​​represented by -127 and the number of values ​​represented by +128.

[0105] If the processing circuitry determines that the number of fixed-point output values ​​represented by one of the extreme values ​​is equal to or less than 1%, then the processing circuitry embeds the current fixed-point weights into a layer of the ANN (step 704). In the context of PTQ process 500, after the processing circuitry determines that the saturation level of the fixed-point output values ​​represents acceptable saturation, the processing circuitry performs step 506.

[0106] Alternatively, if the processing circuit system determines that the number of fixed-point output values ​​represented by one of the extreme values ​​is greater than 1%, then the processing circuit system expands the range of floating-point output values ​​by 2% in the direction of saturation (step 705). In an embodiment, to expand the range of floating-point output values, the processing circuit system first identifies the minimum and maximum fixed-point output values ​​produced by the ANN. Next, the processing circuit system converts the minimum and maximum fixed-point output values ​​to floating-point numbers based on the derived output scaling factor and output zero point of the scaling process 600. For example, the processing circuit system may execute the following equation:

[0107]

[0108]

[0109] So that in equation (1), Y MINfloat Indicates the minimum floating-point output value. The minimum fixed-point value indicates the minimum value allowed by the fixed-point range (for example, -128). Y Indicates output zero, and s Y represents the output scaling factor, and makes Y MAXfloatIndicates the maximum floating-point output value. The maximum fixed-point value indicates the maximum value allowed by the fixed-point range (for example, +127). Y Indicates output zero, and s Y Indicates the output scaling factor.

[0110] Next, the processing circuit system determines the direction of saturation. For example, the processing circuit system may determine which extreme value the fixed-point output value saturates to. If the fixed-point output value saturates toward the low end of the allowable fixed-point range, then the processing circuit system expands the minimum floating-point output value of equation (1) by 2%. Alternatively, if the fixed-point output value saturates toward the high end of the allowable fixed-point range, then the processing circuit system expands the maximum floating-point output value of equation (2) by 2%.

[0111] After updating the fixed-point output range, the processing circuitry returns to the scaling process 600 to recalculate the output scaling factor and the weight scaling factor based on the updated fixed-point output range (step 707). In an embodiment, the processing circuitry iteratively performs the scaling process 600 and the saturation process 700 until the saturation level of the fixed-point output value indicates acceptable saturation. In another embodiment, the processing circuitry iteratively performs the scaling process 600 and the saturation process 700 until a maximum number of calibration iterations has been reached.

[0112] Experimental results related to the disclosed technology show an improvement in the accuracy of neural networks employed via fixed-point hardware. For example, in one experiment, a neural network quantized using the proposed method showed a 16.17% increase in accuracy compared to a neural network quantized using a previous method for performing post-training quantization.

[0113] Figure 8 A results table 800 is shown in an embodiment. Results table 800 represents a table depicting the results for adjusting the weight scaling factor in step 610 of scaling process 600. Results table 800 includes a layer column 801, an output scaling factor column 803, an input scaling factor column 805, a weight scaling factor column 807, a scale ratio column 809, a hardware scale ratio column 811, an adjusted weight scaling factor column 813, and an adjusted scale ratio column 815.

[0114] Layer column 801 stores data for the 91st, 92nd, 93rd, and 94th layers of the ANN. Output scaling factor column 803 stores output scaling factors for the corresponding layers of the ANN. For example, the output scaling factor for the 91st layer is equal to 315.468. Similarly, input scaling factor column 805 and weight scaling factor column 807 store input scaling factors and weight scaling factors for the corresponding layers of the ANN. Scale ratio column 809 stores various scaling ratios for the layers of the ANN, and hardware scaling ratio column 811 stores various hardware scaling ratios. For example, the scaling ratio for the 93rd layer is equal to 0.00097435, and the hardware scaling ratio is equal to 0.000973. Adjusted weight scaling factor column 813 stores adjusted weight scaling factors for the corresponding layers, and adjusted scaling ratio column 815 stores adjusted scaling ratios for the corresponding layers.

[0115] As stated above (ref. Figure 6 ), when adjusting the weight scaling factor, the goal of the processing circuit system is to minimize the difference between the scale ratio and the hardware scale ratio. In an embodiment, the processing circuit system adjusts the weight scaling factor so that the difference between the scale ratio and the hardware scale ratio is less than or equal to 1%. For example, for the 92nd layer of the ANN, the processing circuit system adjusts the weight scaling factor so that the difference between the scale ratio and the hardware scale ratio is reduced from 1.00192% to 0.99991%.

[0116] Fig. 9 Result table 900 is shown in an embodiment. Result table 900 represents a table depicting the results of reducing the number of bits representing hardware bias terms in step 617 of scaling process 600. Result table 900 includes layer column 901, weight scaling factor column 903, hardware bias term column 905, hardware bias bit column 907, adjusted weight scaling factor column 909, adjusted hardware bias term column 911, and adjusted hardware bias bit column 913.

[0117] Layer column 901 stores data for the 2nd, 3rd, 42nd, and 61st layers of the ANN. Weight scaling factor column 903 stores weight scaling factors for the corresponding layers of the ANN. For example, the weight scaling factor for the 2nd layer of the ANN is equal to 1071970624. Hardware bias item column 905 stores hardware bias items for the corresponding layers of the ANN, and hardware bias bit column 907 stores the number of bits used to represent the corresponding hardware bias items. For example, the hardware bias item for the 3rd layer is equal to 3665211682, and the number of bits used to represent the bias item is equal to 32. Adjusted weight scaling factor column 909 stores adjusted weight scaling factors for the corresponding layers of the ANN. Adjusted hardware bias item column 911 stores adjusted hardware bias items for the corresponding layers of the ANN, and adjusted hardware bias bit column 913 stores the number of adjusted hardware bias item bits.

[0118] As stated above (ref. Figure 6 ), when reducing the number of bits representing the hardware bias term, the goal of the processing circuit system is to prevent accumulator overflow errors. For example, if the associated accumulator performs a calculation using 32-bit data, the processing circuit system ensures that the hardware bias term is less than or equal to 30 bits long. In an embodiment, to reduce the number of bits used to represent the hardware bias term, the processing circuit system divides the corresponding weight scaling factor by two. For example, for the 42nd layer of the ANN, the processing circuit system divides the weight scaling factor 3203457 by two to reduce the number of bits representing the hardware bias term from 31 bits to 30 bits.

[0119] Fig.10 A computing system 1001 is shown that represents any system or collection of systems that can implement the various processes, programs, services, and scenarios disclosed herein. The computing system 1001 can be implemented as a single device, system, or apparatus, or can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing system 1001 includes, but is not limited to, a processing system 1002, a storage system 1003, software 1005, a communication interface system 1007, and a user interface system 1009 (optional). The processing system 1002 is operatively coupled to the storage system 1003, the communication interface system 1007, and the user interface system 1009.

[0120] Processing system 1002 loads and executes software 1005 from storage system 1003. Software 1005 includes and implements a post-training quantization (PTQ) process 1006, which represents a process discussed with respect to previous figures, such as PTQ process 103, quantization method 200, quantization logic 303, or PTQ process 500. When executed by processing system 1002, software 1005 directs processing system 1002 to operate as described herein to perform at least the various processes, operational scenarios, and sequences discussed in the foregoing embodiments. Computing system 1001 may optionally include additional devices, features, or functionality that are not discussed for the sake of brevity.

[0121] Still refer to Fig.10 , processing system 1002 includes a microprocessor and other circuitry that retrieves and executes software 1005 from a storage system 1003. Processing system 1002 can be implemented within a single processing device, but can also be distributed across multiple processing devices or subsystems that cooperate to execute program instructions. Examples of processing system 1002 include general-purpose central processing units, graphics processing units, special-purpose processors and logic devices, and any other type of processing device, combination of processing devices, or variations thereof.

[0122] The storage system 1003 includes any computer-readable storage media that can be read by the processing system 1002 and that can store the software 1005. The storage system 1003 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or any other suitable storage media. In any case, a computer-readable storage medium is not a propagated signal.

[0123] In addition to computer-readable storage media, in some embodiments, storage system 1003 may also include computer-readable communication media that can be used to transfer at least some of software 1005 internally or externally. Storage system 1003 can be implemented as a single storage device, but also across multiple storage devices or subsystems that are co-located or distributed relative to each other. Storage system 1003 may include additional elements, such as a controller, that can communicate with processing system 1002 or possibly other systems.

[0124] The software 1005 (including the PTQ process 1006) may be implemented as program instructions and, among other functions, when executed by the processing system 1002, may direct the processing system 1002 to operate as described with respect to the various operational scenarios, sequences, and processes shown herein. For example, the software 1005 may include program instructions for implementing the process described herein for identifying a scaling factor. The computing system 1001 may be coupled to one or more sensors and configured to receive input from one or more sensors (e.g., a camera, radar, lidar, microphone, etc.). The sensor input may form the basis of the input data in the PTQ process 1006.

[0125] In particular, program instructions may include various components or modules that collaborate or otherwise interact to perform various processes and operating scenarios described herein. Various components or modules may be embodied in compiling or interpreting instructions, or in some other variation or combination of instructions. Various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single-threaded environment or multithreaded, or according to any other suitable execution paradigm, variation or combination thereof. Software 1005 may include additional processes, programs or components, such as operating system software, virtualization software or other application software. Software 1005 may also include firmware or some other form of machine-readable processing instructions that may be executed by processing system 1002.

[0126] In general, software 1005, when loaded into processing system 1002 and executed, can transform the appropriate device, system, or apparatus (represented by computing system 1001) from a general purpose computing system to a special purpose computing system that is customized to support the execution of the reasoning model in an optimized manner. The encoded software 1005 on storage system 1003 can transform the physical structure of storage system 1003. In different embodiments of the present specification, the specific transformation of the physical structure can depend on various factors. Examples of such factors can include, but are not limited to, the technology used to implement the storage media of storage system 1003 and whether the computer storage media is characterized as a primary storage device or as a secondary storage device, as well as other factors.

[0127] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, the software 1005 may transform the physical state of the semiconductor memory when the program instructions are encoded therein, for example by transforming the states of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of the physical media are possible without departing from the scope of this specification, with the foregoing examples provided merely to facilitate this discussion.

[0128] The communication interface system 1007 may include communication connections and devices that allow communication with other computing systems (not shown) over a communication network (not shown). Examples of connections and devices that together allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over a communication medium such as metal, glass, air, or any other suitable communication medium for exchanging communications with other computing systems or systems networks. The foregoing media, connections, and devices are well known and need not be discussed in detail here.

[0129] Communications between computing system 1001 and other computing systems (not shown) may be conducted over one or more communications networks and according to various communications protocols, combinations of protocols, or variations thereof. Examples include intranets, interconnects, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The aforementioned communications networks and protocols are well known and need not be discussed in detail here.

[0130] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Thus, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects (all of which may be collectively referred to herein as a "circuit," "module," or "system"). Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0131] It will be appreciated that while the inventive concepts disclosed herein are discussed in the context of such productivity applications, the inventive concepts are also applicable to other contexts, such as gaming applications, virtual and augmented reality applications, enterprise applications, and other types of software applications. Likewise, the concepts are applicable not only to electronic documents, but also to other types of content, such as in-game electronic content, virtual and augmented content, databases, language / text, and audio and video content.

[0132] In fact, the included descriptions and figures depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching the inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will understand variations within the scope of the present disclosure from these embodiments. Those skilled in the art will also understand that the features described above can be combined in various ways to form multiple embodiments. Therefore, the present invention is not limited to the specific embodiments described above, but is limited only by the claims and their equivalents.

Claims

1. A system comprising: Memory; as well as a processor coupled to the memory and configured to: a) obtaining a fixed-point output value from a layer of an artificial neural network ANN, wherein the layer comprises fixed-point weights determined based on floating-point weights and a weight scaling factor determined based on an output scaling factor; b) converting the fixed-point output value to a floating-point output value based on the output scaling factor; c) Expand the range of floating point output values; as well as d) calculating a new output scaling factor based on the extended range of floating point output values; as well as e) Storing the new output scaling factor in the memory.

2. The system of claim 1 , wherein before the processor converts the fixed-point output value to a floating-point output value based on the output scaling factor in step (b), the processor is further configured to determine whether a saturation level of the fixed-point output value includes acceptable saturation.

3. The system of claim 2, wherein to determine that the saturation level comprises the acceptable saturation, the processor is further configured to accept the fixed-point weights and avoid performing steps (b) through (e).

4. The system of claim 2, wherein to determine that the saturation level does not include the acceptable saturation, the processor is further configured to proceed with steps (b) through (e).

5. The system of claim 4, wherein after the processor stores the new output scaling factor in the memory in step (e), the processor is further configured to: f) calculating a new weight scaling factor based on the new output scaling factor; g) updating the fixed-point weights based on the new weight scaling factor; and h) Return to step (a).

6. The system of claim 5, wherein to calculate the new weight scaling factor in step (f), the processor is further configured to: calculating an ideal weight scaling factor based on a hardware scale, wherein the hardware scale represents a ratio of a hardware scaling factor to a hardware shift factor; determining a difference between the weight scaling factor and the ideal weight scaling factor; and The new weight scaling factor is determined based on the difference between the weight scaling factor and the ideal weight scaling factor.

7. The system of claim 1, wherein to obtain the fixed-point output value in step (a), the processor is further configured to input a fixed-point input value to the ANN configured with the fixed-point weights to generate the fixed-point output value.

8. The system of claim 1 , wherein before the processor obtains the fixed-point output value in step (a), the processor is further configured to determine the fixed-point weight, and wherein to determine the fixed-point weight, the processor is further configured to scale the floating-point weight based on the weight scaling factor.

9. The system of claim 8, wherein the processor is further configured to determine the weight scaling factor, and wherein to determine the weight scaling factor, the processor is further configured to: inputting a floating point input value into the ANN configured with the floating point weights to obtain an initial floating point output value; calculating an input scaling factor based on the floating point input value; calculating the output scaling factor based on the initial floating point output value; calculating the weight scaling factor based on the floating point weight; as well as The weight scaling factor is adjusted based on an allowable number of hardware bias entry bits.

10. A method comprising: a) obtaining a fixed-point output value from a layer of an artificial neural network ANN, wherein the layer comprises fixed-point weights determined based on floating-point weights and a weight scaling factor determined based on an output scaling factor; b) converting the fixed-point output value to a floating-point output value based on the output scaling factor; c) Expand the range of floating point output values; as well as d) Calculating a new output scaling factor based on the extended range of floating point output values.

11. The method of claim 10, further comprising determining whether a saturation level of the fixed-point output value includes acceptable saturation before converting the fixed-point output value to the floating-point output value based on the output scaling factor in step (b).

12. The method according to claim 11, further comprising: In response to determining that the saturation level includes the acceptable saturation, accepting the fixed-point weights and refraining from performing steps (b) through (d); as well as In response to determining that the saturation levels do not include the acceptable saturation, proceeding with steps (b) through (d).

13. The method of claim 10, further comprising performing the following operations after calculating the new output scaling factor in step (d): e) calculating a new weight scaling factor based on the new output scaling factor; f) updating the fixed-point weights based on the new weight scaling factor; and g) Return to step (a).

14. The method of claim 13, wherein calculating the new weight scaling factor in step (e) comprises: calculating an ideal weight scaling factor based on a hardware scale, wherein the hardware scale represents a ratio of a hardware scaling factor to a hardware shift factor; determining a difference between the weight scaling factor and the ideal weight scaling factor; as well as The new weight scaling factor is determined based on the difference between the weight scaling factor and the ideal weight scaling factor.

15. The method of claim 10, wherein obtaining the fixed-point output value in step (a) comprises inputting a fixed-point input value to the ANN configured with the fixed-point weights to generate the fixed-point output value.

16. The method of claim 10, further comprising determining the fixed-point weight before obtaining the fixed-point output value in step (a), wherein determining the fixed-point weight comprises scaling the floating-point weight based on the weight scaling factor.

17. The method of claim 16, further comprising determining the weight scaling factor, wherein determining the weight scaling factor comprises: inputting a floating point input value into the ANN configured with the floating point weights to obtain an initial floating point output value; calculating an input scaling factor based on the floating point input value; calculating the output scaling factor based on the initial floating point output value; calculating the weight scaling factor based on the floating point weight; as well as The weight scaling factor is adjusted based on an allowable number of hardware bias entry bits.

18. The method of claim 17, wherein calculating the new output scaling factor in step (d) comprises: determining the allowable number of fixed-point output values; as well as The allowable number of fixed-point output values ​​is divided by the extended range of floating-point output values.

19. One or more computer-readable storage media having program instructions stored thereon that, when executed by one or more processors, direct a computing device to perform at least the following operations: a) obtaining a fixed-point output value from a layer of an artificial neural network ANN, wherein the layer comprises fixed-point weights determined based on floating-point weights and a weight scaling factor determined based on an output scaling factor; b) converting the fixed-point output value to a floating-point output value based on the output scaling factor; c) Expand the range of floating point output values; as well as d) Calculating a new output scaling factor based on the extended range of floating point output values.

20. A processing circuit system coupled with stored instructions for implementing a method, wherein the instructions, when executed by the processing circuit system, perform the steps comprising: a) obtaining a fixed-point output value from a layer of an artificial neural network ANN, wherein the layer comprises fixed-point weights determined based on floating-point weights and a weight scaling factor determined based on an output scaling factor; b) converting the fixed-point output value to a floating-point output value based on the output scaling factor; c) Expand the range of floating point output values; as well as d) Calculating a new output scaling factor based on the extended range of floating point output values.

Images