Dynamic quantification method and device, equipment, storage medium and program product

Abstract

The invention relates to a dynamic quantification method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a first maximum value in a to-be-quantized set, wherein the first maximum value is the maximum value of absolute values of a plurality of to-be-quantized data in the to-be-quantized set; determining a quantization parameter according to the first maximum value and a second maximum value, wherein the second maximum value is the maximum value which can be represented by the quantization target data type; and performing quantization processing on the to-be-quantized data according to the quantization parameters to obtain quantized data corresponding to the to-be-quantized data. By adopting the method, the neural network processing unit can directly obtain the quantized activation data or weight data without pre-generating and transmitting massive quantized data and quantized parameters through an off-line quantization process, so that the occupation of storage resources in the reasoning process can be effectively reduced, and the bandwidth consumption of data transmission is reduced at the same time.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a dynamic quantization method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of deep learning technology, the complexity and scale of models are constantly increasing. Traditional high-precision models face significant challenges in terms of storage, computation speed, and energy consumption. Quantization techniques can map data in neural network models from high-precision data types (such as float32, float16) to low-precision data types (such as int8, int4, float8, float4), thereby reducing data storage requirements, computational overhead, and energy consumption by lowering computational precision. This has become an important technique for deploying and optimizing neural network models.

[0003] In existing technologies, quantization techniques involve offline quantization of the weight data or activation data of a neural network model. For example, a separate quantization tool is needed to quantize the weight data or activation data after training, or quantization is performed during model training.

[0004] However, the aforementioned offline quantization techniques require transmitting the quantized data to the neural network processing unit for subsequent computation. When the number of weight parameters or activation parameters in the neural network model is excessive, a large amount of data needs to be transmitted to the neural network processing unit, resulting in significant consumption of system storage resources and bandwidth. Summary of the Invention

[0005] Therefore, it is necessary to provide a dynamic quantization method, apparatus, device, storage medium, and program product that can reduce storage resource and bandwidth usage in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a dynamic quantization method applied to a neural network processing unit, comprising:

[0007] Find the first maximum value in the set to be quantized. The first maximum value is the maximum of the absolute values ​​of multiple data points to be quantized in the set.

[0008] The quantization parameters are determined based on the first and second maximum values, where the second maximum value is the maximum value that the target data type can represent.

[0009] The data to be quantized is quantized according to the quantization parameters to obtain the quantized data corresponding to each data to be quantized.

[0010] In one embodiment, determining the quantization parameter based on a first maximum value and a second maximum value includes:

[0011] Determine the first ratio between the first maximum value and the second maximum value;

[0012] The quantization parameters are determined based on the first ratio.

[0013] In one embodiment, each piece of data to be quantized is quantized according to quantization parameters to obtain quantized data corresponding to each piece of data to be quantized, including:

[0014] For each data to be quantized, determine the second ratio between the data to be quantized and the quantization parameter;

[0015] If the second ratio is within the target data range, the second ratio is rounded or truncated to obtain the quantized data. The target data range is the range of data that the target data type can represent.

[0016] If the second ratio is not within the target data range, the quantized data is determined from the boundary values ​​of the target data range.

[0017] In one embodiment, the data types of the data to be quantized and the first ratio are floating-point data, including sign bits, exponent bits, and mantissa bits. The quantization parameters are determined based on the first ratio, including:

[0018] When all the last digits of the first ratio are 0, the exponent of the first ratio is determined as the quantization parameter;

[0019] When the last digits of the first ratio are not all zero, the sum of the exponent of the first ratio and the preset value is used as the quantization parameter.

[0020] In one embodiment, determining a second ratio of the data to be quantized to the quantization parameter includes:

[0021] The offset value corresponding to the exponent is determined based on the bit width of the exponent of the data to be quantized.

[0022] Determine the difference between the exponent of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent.

[0023] The second ratio is obtained by concatenating the sign bit, the target exponent bit, and the mantissa bit of the data to be quantized.

[0024] In one embodiment, the method further includes:

[0025] Obtain multiple data points to be quantified;

[0026] Multiple data sets to be quantized are grouped according to a preset number of elements to obtain multiple sets to be quantized.

[0027] Secondly, this application also provides a dynamic quantization device applied to a neural network processing unit, comprising:

[0028] The acquisition module is used to acquire the first maximum value in the set to be quantized. The first maximum value is the maximum of the absolute values ​​of multiple data to be quantized in the set to be quantized.

[0029] The determination module is used to determine the quantization parameters based on the first maximum value and the second maximum value, where the second maximum value is the maximum value that the target data type can represent.

[0030] The quantization module is used to quantize each data to be quantized according to the quantization parameters, so as to obtain the quantized data corresponding to each data to be quantized.

[0031] In one embodiment, the determining module is specifically used to determine a first ratio between the first maximum value and the second maximum value; and to determine quantization parameters based on the first ratio.

[0032] In one embodiment, the quantization module is specifically used to determine a second ratio between the data to be quantized and the quantization parameter for each data to be quantized; if the second ratio is within the target data range, rounding or truncating the second ratio to obtain quantized data, wherein the target data range is the data range that the target data type can represent; if the second ratio is not within the target data range, determining the quantized data from the boundary values ​​of the target data range.

[0033] In one embodiment, the data types of the data to be quantized and the first ratio are floating-point data, including a sign bit, an exponent bit, and a mantissa bit. The determining module is specifically used to determine the exponent bit of the first ratio as a quantization parameter when all the mantissa bits of the first ratio are 0; and to use the sum of the exponent bit of the first ratio and a preset value as a quantization parameter when not all the mantissa bits of the first ratio are 0.

[0034] In one embodiment, the quantization module is specifically used to determine the offset value corresponding to the exponent bit based on the bit width of the exponent bit of the data to be quantized; determine the difference between the exponent bit of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent bit; and concatenate the sign bit, the target exponent bit, and the mantissa bit of the data to be quantized to obtain a second ratio.

[0035] In one embodiment, the acquisition module is further configured to acquire multiple data to be quantized; and group the multiple data to be quantized according to a preset number of elements to obtain multiple sets of data to be quantized.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in any of the first aspects above.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects above.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0039] In a sixth aspect, this application also provides a chip, including a processor, for calling and running a computer program from a memory, causing all devices equipped with the chip to perform the steps of the method described in any of the first aspects above.

[0040] The aforementioned dynamic quantization method, apparatus, device, storage medium, and program product obtain a first maximum value in the set to be quantized, which is the maximum absolute value of multiple data to be quantized in the set; determine quantization parameters based on the first and second maximum values, where the second maximum value is the maximum value that the target data type can represent; and quantize each data to be quantized according to the quantization parameters to obtain the quantized data corresponding to each data to be quantized. Thus, when the neural network processing unit performs convolution operations and needs to quantize activation data or weight data, it can first obtain the first maximum value of the input activation data or weight data, then determine the quantization parameters based on the first maximum value and the second maximum value that the target data type can represent, and then perform online dynamic quantization of the activation data or weight data according to the determined quantization parameters. During the model inference stage, the neural network processing unit can directly obtain the quantized activation data or weight data without having to pre-generate and transmit massive amounts of quantized data and quantization parameters through an offline quantization process, effectively saving storage resources during inference and reducing bandwidth consumption during data transmission. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the application environment of the dynamic quantization method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating a dynamic quantization method in one embodiment;

[0044] Figure 3This is a flowchart illustrating the dynamic quantization method in another embodiment;

[0045] Figure 4(a) is a schematic diagram of the grouping of data to be quantified in one embodiment;

[0046] Figure 4(b) is a schematic diagram of the grouping of data to be quantified in another embodiment;

[0047] Figure 5 This is a flowchart illustrating the steps for determining quantization parameters in one embodiment;

[0048] Figure 6 This is a flowchart illustrating the steps for determining the quantized data corresponding to the data to be quantized in one embodiment.

[0049] Figure 7 This is a schematic diagram of the process for determining the second ratio of the data to be quantized to the quantization parameter in one embodiment;

[0050] Figure 8 This is a flowchart illustrating the dynamic quantization method in another embodiment;

[0051] Figure 9 This is a flowchart illustrating the dynamic quantization method in another embodiment;

[0052] Figure 10 This is a structural block diagram of the dynamic quantization device in one embodiment;

[0053] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0056] The dynamic quantization method provided in this application embodiment can be applied to, for example, Figure 1The neural network processing unit (NPU) shown is described. The NPU 10 includes a control unit 101, a storage unit 102, and a dynamic quantization unit 103. The input to the NPU 10 is high-precision floating-point weight data or activation data. The control unit 101 acquires this high-precision floating-point weight data or activation data from an external source according to certain constraints and order, and stores it in the storage unit 102. Simultaneously, the control unit 101 also controls the storage unit to output the high-precision floating-point weight data or activation data to the dynamic quantization unit 103 according to certain constraints and order. The dynamic quantization unit 103 acquires the first maximum value in the set to be quantized, then determines the quantization parameters based on the first maximum value and the second maximum value that the target data type can represent. Finally, it quantizes each set of data to be quantized according to the quantization parameters to obtain the quantized data corresponding to each set of data. After processing by the dynamic quantization unit 103, the weight data or activation data becomes integer or low-precision floating-point data. The dynamic quantization unit 103 outputs the quantized data and the quantization parameters.

[0057] In one exemplary embodiment, such as Figure 2 As shown, a dynamic quantization method is provided, including the following steps 201 to 203. Wherein:

[0058] Step 201: Obtain the first maximum value in the set to be quantized. The first maximum value is the maximum absolute value of multiple data to be quantized in the set to be quantized.

[0059] The data to be quantified can be either activation data or weighted data; this application does not limit this.

[0060] Quantization can be the process of mapping data to be quantized from a high-precision data type (such as float32, float16) to a low-precision data type (such as int8, int4, float8, float4). FP16 can be used to represent float16, and FP32 can be used to represent float32. The representation of floating-point values ​​(half-precision floating-point numbers float16, single-precision floating-point numbers float32, etc.) must conform to the IEEE 754 standard, that is, using a sign bit, exponent bit, and mantissa bit to represent floating-point data.

[0061] An IEEE 754 format floating-point data, such as float16, consists of 16 bits of binary data, where

[15] bits represent the sign bit, [14:10] bits represent the exponent bits, and [9:0] bits represent the mantissa bits.

[0062] Optionally, in this embodiment, the neural network model can be a model for processing images, videos, speech, or text, such as a classification model, an object detection model, or a Large Language Model (LLM) model. The data to be quantized can be the weight data in the aforementioned neural network model for processing images, videos, speech, or text, or the activation data of the aforementioned neural network model during the inference process.

[0063] For example, in the process of image classification, quantizing the data type of the weight data and activation data of the image classification model from FP32 to int8 can reduce the size of the image classification model and improve the inference speed.

[0064] Activation data can be the output values ​​generated by neurons in each layer of a neural network when processing input data; it is also called activation values ​​or feature maps. Taking the ResNet image classification model as an example, ResNet includes an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. In the input layer, activation data can be the pixel tensor of the original image input to the ResNet model; in the convolutional layers, activation data can be the image features extracted by each convolutional kernel, and so on.

[0065] The set of data to be quantized processed by the neural network processing unit may include activation data or weight data required for a single convolution calculation, or it may be that the activation data or weight data required for a single convolution calculation are grouped according to a preset number to obtain multiple sets of data to be quantized. Then, the data to be quantized in each set of data to be quantized is quantized according to steps 201 to 203. This application embodiment does not limit this.

[0066] For a set to be quantized, determine the absolute values ​​of multiple data points to be quantized in the set, and then determine the maximum value among the multiple absolute values ​​as the first maximum value.

[0067] Step 202: Determine the quantization parameters based on the first maximum value and the second maximum value. The second maximum value is the maximum value that the target data type can represent.

[0068] Optionally, the target data type for quantization can be a low-precision integer or a specially compressed floating-point type after converting the original high-precision floating-point data (such as FP32). The target data type for quantization can be int8, int4, float8, or float4, etc.

[0069] When the target data type for quantization is int8, the second maximum value can be 127; when the target data type for quantization is int4, the second maximum value is 7. These details will not be elaborated further here.

[0070] The quantization parameter can be determined based on the first ratio of the first maximum value to the second maximum value. The first ratio of the first maximum value to the second maximum value can be determined directly by a divider or by an adder. This application does not limit the specific method used.

[0071] The quantization parameter can be a first ratio or a first ratio obtained by processing it; this application does not limit this.

[0072] Step 203: Quantize each data to be quantized according to the quantization parameters to obtain the quantized data corresponding to each data to be quantized.

[0073] Optionally, for each data point in the set to be quantized, after determining the quantization parameters corresponding to the set to be quantized, the corresponding quantized data can be determined based on each data point and the quantization parameters.

[0074] For example, a second ratio between the data to be quantized and the quantization parameters can be determined based on the divider, and then the quantized data corresponding to the data to be quantized can be determined based on the second ratio.

[0075] Optionally, the second ratio of the data to be quantized to the quantization parameter may not be within the range of target data that the target data type can represent. Therefore, when the second ratio is not within the range of target data, it is necessary to clamp the second ratio to obtain the corresponding quantized data.

[0076] The aforementioned dynamic quantization method obtains the first maximum value in the set to be quantized, which is the maximum absolute value of multiple data points in the set. Quantization parameters are then determined based on the first and second maximum values, where the second maximum value is the maximum value that the target data type can represent. Each data point is then quantized according to these parameters to obtain the quantized data. Thus, when the neural network processing unit performs convolution operations and needs to quantize activation or weight data, it can first obtain the first maximum value of the input activation or weight data. Then, based on the first maximum value and the second maximum value that the target data type can represent, the quantization parameters are determined. The determined quantization parameters are then used to perform online dynamic quantization of the activation or weight data. During the model inference stage, the neural network processing unit can directly obtain the quantized activation or weight data without needing to pre-generate and transmit massive amounts of quantized data and parameters through an offline quantization process. This effectively saves storage resources during inference and reduces bandwidth consumption during data transmission.

[0077] In one exemplary embodiment, such as Figure 3 As shown, the dynamic quantization method further includes steps 301 to 302. Wherein:

[0078] Step 301: Obtain multiple data points to be quantified.

[0079] In this context, multiple data points to be quantized can be either activation data or weight data required for a single convolution calculation. The NPU requires both activation and weight data for convolution calculations, and the amount of data processed each time is fixed. Therefore, the amount of data output by the activation data processing unit each time is also fixed, which can be represented by the parameter `conv_width`.

[0080] Step 302: Group the multiple data to be quantized according to the preset number of elements to obtain multiple sets to be quantized.

[0081] Optionally, the activation data or weight data required for a single convolution calculation can be grouped according to a preset number of elements to obtain multiple sets to be quantized.

[0082] For example, if each group has group_size high-precision data to be quantized, then for the conv_width data to be quantized required for one convolution calculation, it can be divided into a group = conv_width / group_size set of data to be quantized.

[0083] Then, for the data to be quantized in each set of data to be quantized, group quantization is performed, and the quantization parameters in each set of data to be quantized are determined. Then, the data to be quantized in the set of data to be quantized is quantized according to the quantization parameters corresponding to each set of data to be quantized.

[0084] For example, taking the dynamic quantization of FP16 floating-point activation data into int8 as an example, that is, the data type of the data to be quantized is FP16, and the data type of the quantization target is int8. Assuming that one convolution calculation requires the neural network processing unit (NPU) to output conv_width=128 data points at a time, for a 1x1 convolution window, the amount of data output by the NPU at one time is shown in Figure 4(a), which can correspond to coordinates (x,y,p) as (0,0,0)~(0,0,127), i.e., labeled ①②③④, a total of 128 data points. The next NPU... The amount of data output by the PU can correspond to 128 data points with coordinates (x,y,p) ranging from (1,0,0) to (1,0,127). For a 2x2 convolution window, the amount of data output by the NPU in one operation is shown in Figure 4(b), which corresponds to 128 data points with coordinates (x,y,p) ranging from (0,0,0) to (1,1,31), labeled ①②③④. The amount of data output by the NPU in the next operation can correspond to 128 data points with coordinates (x,y,p) ranging from (0,0,32) to (1,1,63). The same logic applies to an nxn convolution window.

[0085] Assuming the preset group size is 32, the 128 data points in Figures 4(a) and 4(b) can be divided into 4 groups, corresponding to labels ①, ②, ③, and ④ respectively. Dynamic quantization of a set of 32 FP16 format data points can determine a quantization parameter and 32 quantized int8 format data points.

[0086] By grouping the data to be quantized, when performing dynamic quantization of the data to be quantized in steps 201 to 203, each group can be configured with quantization parameters independently. This allows for more accurate adaptation to the distribution characteristics of each group of data and avoids the situation where each layer or tensor has only one quantization parameter. This avoids the loss of precision in activation data or weight data and improves quantization accuracy.

[0087] In one exemplary embodiment, such as Figure 5 As shown, the quantization parameters are determined based on the first and second maximum values, including steps 501 to 502. Wherein:

[0088] Step 501: Determine the first ratio between the first maximum value and the second maximum value.

[0089] Alternatively, the first ratio of the first maximum value to the second maximum value can be calculated directly using a divider. A floating-point multiplier can also be used instead of a divider; for example, when determining the first ratio of the first maximum value to the second maximum value, the first ratio can also be expressed as... ,in, The first maximum value, The second maximum value is obtained by comparing the first maximum value with... The product of the two values ​​determines the first ratio.

[0090] To save resources, adders can be used instead of dividers; for example, adders can be used instead of dividers. It is approximately represented as a power of 2. Then, the exponent of the first maximum value is determined, and the exponent of the first maximum value is added to the power of 2 to determine the first ratio.

[0091] For example, when the data format of the data to be quantized is FP16, the set to be quantized includes 32 data points to be quantized, and the data type of the quantization target is int8, the first maximum value is... The second maximum value is 127, so 1 / (N_leves-1) = 1 / 127. In FP16 format, this is represented as ceil(1 / 127) = 0x2009, which needs to be rounded up. The first ratio can be expressed as:

[0092]

[0093] Step 502: Determine the quantization parameters based on the first ratio.

[0094] In one possible implementation, the first ratio can be directly used as the quantization parameter.

[0095] In another possible implementation, the data to be quantized and the first ratio are of floating-point data, including sign bits, exponent bits, and mantissa bits. The quantization parameters are determined based on the first ratio, including:

[0096] When all the last digits of the first ratio are 0, the exponent of the first ratio is determined as the quantization parameter; when not all the last digits of the first ratio are 0, the sum of the exponent of the first ratio and the preset value is used as the quantization parameter.

[0097] Optionally, in the neural network processing unit, the above calculations are all in binary form, so the exponent and mantissa of the first ratio can be obtained.

[0098] The preset value can be 1.

[0099] The quantization parameters can be determined using the following expression:

[0100]

[0101] Where scale represents the quantization parameter. This indicates the exponent of the first ratio. This indicates the last digit of the first ratio. () is a bitwise OR reduction function that performs an OR operation on all the bits of the mantissa of the first ratio.

[0102] For example, taking the data format of the first ratio as FP16, the exponent of the first ratio is the [14:10] bits of the first ratio, and the mantissa of the first ratio is the [9:0] bits of the first ratio. In this case, the quantization parameters can be determined according to the following expression:

[0103]

[0104] in, This indicates the exponent of the first ratio. This indicates the last digit of the first ratio. This indicates that the 10 bits of the mantissa are not all 0. In this case, the sum of the exponent of the first ratio and 1 is used as the quantization parameter. This indicates that all 10 bits of the mantissa are 0. In this case, the exponent of the first ratio is used as the quantization parameter.

[0105] The first ratio between the first maximum value and the second maximum value is determined above. Based on the first ratio, the quantization parameters are determined. Subsequently, the activation data or weight data can be dynamically quantized online based on the determined quantization parameters. In this way, during the model inference stage, the neural network processing unit can directly obtain the quantized activation data or weight data without having to pre-generate and transmit massive amounts of quantized data and quantization parameters through an offline quantization process. This can effectively save storage resources occupied during the inference process and reduce bandwidth consumption for data transmission.

[0106] In one exemplary embodiment, such as Figure 6 As shown, the data to be quantized is quantized according to the quantization parameters to obtain the quantized data corresponding to each data point, including the following steps 601 to 603. Wherein:

[0107] Step 601: For each data to be quantized, determine the second ratio between the data to be quantized and the quantization parameter.

[0108] For each data point in the set to be quantized, quantization can be performed based on the quantization parameters.

[0109] In one possible implementation, the second ratio of the data to be quantized to the quantization parameter can be directly calculated based on the divider, that is, the second ratio = data to be quantized / quantization parameter.

[0110] In another possible implementation, to save resources, an adder can be used instead of a divider. In this case, the quantization parameter can be the exponent of the first ratio, or it can be the sum of the exponent of the first ratio and a preset value. For a given set of data to be quantized, a second ratio between the data to be quantized and the quantization parameter is determined, such as... Figure 7 As shown, the process includes steps 701 to 703. Wherein:

[0111] Step 701: Determine the offset value corresponding to the exponent bit based on the bit width of the exponent bit of the data to be quantized.

[0112] In floating-point encoding standards (such as IEEE 754), the offset value corresponding to the exponent bit is a constant set to convert the signed exponent into an unsigned integer for storage. Its core function is to prevent the positive and negative sign bits of the exponent from occupying extra space.

[0113] Alternatively, the offset value can be determined based on the bit width of the exponent of the data to be quantized, specifically according to the following formula:

[0114]

[0115] Where bias represents the offset value and k represents the exponent bit width.

[0116] For example, when the data type of the data to be quantized is FP16, the exponent bit width is 5 and the offset value is 15. When the data type of the data to be quantized is FP32, the exponent bit width is 8 and the offset value is 127.

[0117] Step 702: Determine the difference between the exponent of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent.

[0118] Optionally, the exponent of the data to be quantized is obtained. Since the quantization parameter at this time is the exponent of the second ratio, the difference between the exponent of the data to be quantized and the quantization parameter can be calculated based on the adder, and then the sum of the difference and the offset value is used as the target exponent.

[0119] Taking the data to be quantized as FP16 as an example, with an offset value of 15, the target exponent can be represented as: .

[0120] Step 703: Concatenate the sign bit, target exponent bit, and mantissa bit of the data to be quantized to obtain the second ratio.

[0121] Optionally, the exponent of the data to be quantized can be replaced with the target exponent to obtain a second ratio.

[0122] Taking the data to be quantized as FP16 as an example, the second ratio can be expressed as: .

[0123] in, The sign bit of the data to be quantized. For the target exponent, The last few digits of the data to be quantified.

[0124] Step 602: If the second ratio is within the target data range, round or truncate the second ratio to obtain the quantized data.

[0125] Step 603: If the second ratio is not within the target data range, determine the quantized data from the boundary values ​​of the target data range.

[0126] The target data range refers to the range of data that the target data type can represent.

[0127] Optionally, the second ratio can be used to force the data to be quantized from the source data type to the target data type. It is necessary to determine whether the second ratio overflows, that is, whether the second ratio is within the target data range.

[0128] When the second ratio falls within the target data range, it can be rounded or truncated to ensure the precision meets the target data type for quantization. In this case, the quantized data retains its numerical semantics, but the precision decreases.

[0129] For example, when the second ratio is FP16 and the quantization target data type is int8, the quantized data of the data to be quantized can be obtained by truncating or rounding the second ratio to a decimal place.

[0130] When the second ratio is not within the target data range, that is, when the second ratio is greater than the maximum value of the target data range or less than the minimum value of the target data range, the quantization data can be determined based on the boundary value of the target data range.

[0131] For example, when the second ratio is greater than the maximum value of the target data range, the maximum value of the target data range can be used as the quantized data after the data to be quantized. When the second ratio is less than the minimum value of the target data range, the minimum value of the target data range can be used as the quantized data after the data to be quantized.

[0132] For each piece of data to be quantized, a second ratio between the data to be quantized and the quantization parameters is determined. If the second ratio is within the target data range, it is rounded or truncated to obtain the quantized data. If the second ratio is not within the target data range, the quantized data is determined from the boundary values ​​of the target data range. During the model inference stage, the neural network processing unit can directly obtain the quantized activation data or weight data without having to pre-generate and transmit massive amounts of quantized data and quantization parameters through an offline quantization process. This effectively saves storage resources during the inference process and reduces bandwidth consumption for data transmission.

[0133] As an optional implementation method, such as Figure 8 As shown, the dynamic quantization method provided in this application embodiment may include the following specific steps:

[0134] Step 801: Obtain multiple data points to be quantified;

[0135] Step 802: Group the multiple data to be quantized according to the preset number of elements to obtain multiple sets to be quantized;

[0136] Step 803: For each set to be quantized, obtain the first maximum value in the set to be quantized. The first maximum value is the maximum absolute value of multiple data to be quantized in the set to be quantized.

[0137] Step 804: Determine the first ratio between the first maximum value and the second maximum value, where the second maximum value is the maximum value that the target data type can represent;

[0138] Step 805: Determine the first ratio as the quantization parameter;

[0139] Step 806: For each data to be quantized, determine the second ratio between the data to be quantized and the quantization parameter;

[0140] Step 807: If the second ratio is within the target data range, round or truncate the second ratio to obtain the quantized data;

[0141] Step 808: If the second ratio is not within the target data range, determine the quantized data from the boundary values ​​of the target data range;

[0142] The target data range refers to the range of data that the target data type can represent.

[0143] As an optional implementation method, such as Figure 9 As shown, the dynamic quantization method provided in this application embodiment may include the following specific steps:

[0144] Step 901: Obtain multiple data points to be quantified;

[0145] Step 902: Group the multiple data to be quantized according to the preset number of elements to obtain multiple sets to be quantized;

[0146] Step 903: For each set to be quantized, obtain the first maximum value in the set to be quantized. The first maximum value is the maximum absolute value of multiple data to be quantized in the set to be quantized.

[0147] Step 904: Determine the first ratio between the first maximum value and the second maximum value, where the second maximum value is the maximum value that the target data type can represent;

[0148] Among them, the data to be quantized and the first ratio are of floating-point data, including sign bit, exponent bit and mantissa bit;

[0149] Step 905: When all the last digits of the first ratio are 0, the exponent of the first ratio is determined as the quantization parameter;

[0150] Step 906: When the last digits of the first ratio are not all zero, use the sum of the exponent of the first ratio and the preset value as the quantization parameter.

[0151] Step 907: Determine the offset value corresponding to the exponent bit based on the bit width of the exponent bit of the data to be quantized;

[0152] Step 908: Determine the difference between the exponent of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent;

[0153] Step 909: Concatenate the sign bit, target exponent bit, and mantissa bit of the data to be quantized to obtain the second ratio;

[0154] Step 910: If the second ratio is within the target data range, round or truncate the second ratio to obtain the quantized data;

[0155] Step 911: If the second ratio is not within the target data range, determine the quantized data from the boundary values ​​of the target data range;

[0156] The target data range refers to the range of data that the target data type can represent.

[0157] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0158] Based on the same inventive concept, this application also provides a dynamic quantization apparatus for implementing the dynamic quantization method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more dynamic quantization apparatus embodiments provided below can be found in the limitations of the dynamic quantization method described above, and will not be repeated here.

[0159] In one exemplary embodiment, such as Figure 10 As shown, a dynamic quantization device 1000 is provided, including: an acquisition module 1001, a determination module 1002, and a quantization module 1003, wherein:

[0160] The acquisition module 1001 is used to acquire the first maximum value in the set to be quantized, which is the maximum value of the absolute values ​​of multiple data to be quantized in the set to be quantized;

[0161] The determination module 1002 is used to determine the quantization parameters based on the first maximum value and the second maximum value, wherein the second maximum value is the maximum value that the target data type can represent.

[0162] The quantization module 1003 is used to quantize each data to be quantized according to the quantization parameters, so as to obtain the quantized data corresponding to each data to be quantized.

[0163] In one embodiment, the determining module 1002 is specifically used to determine a first ratio between the first maximum value and the second maximum value; and to determine quantization parameters based on the first ratio.

[0164] In one embodiment, the quantization module 1003 is specifically used to determine a second ratio between the data to be quantized and the quantization parameter for each data to be quantized; if the second ratio is within the target data range, rounding or truncating the second ratio to obtain quantized data, wherein the target data range is the data range that the target data type can represent; if the second ratio is not within the target data range, determining the quantized data from the boundary values ​​of the target data range.

[0165] In one embodiment, the data types of the data to be quantized and the first ratio are floating-point data, including a sign bit, an exponent bit, and a mantissa bit. The determining module 1002 is specifically used to determine the exponent bit of the first ratio as a quantization parameter when all the mantissa bits of the first ratio are 0; and to use the sum of the exponent bit of the first ratio and a preset value as a quantization parameter when not all the mantissa bits of the first ratio are 0.

[0166] In one embodiment, the quantization module 1003 is specifically used to determine the offset value corresponding to the exponent bit based on the bit width of the exponent bit of the data to be quantized; determine the difference between the exponent bit of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent bit; and concatenate the sign bit, the target exponent bit and the mantissa bit of the data to be quantized to obtain a second ratio.

[0167] In one embodiment, the acquisition module 1001 is further configured to acquire multiple data to be quantized; and group the multiple data to be quantized according to a preset number of elements to obtain multiple sets of data to be quantized.

[0168] Each module in the aforementioned dynamic quantization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0169] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to dynamic quantization. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a dynamic quantization method.

[0170] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described in any of the above method embodiments.

[0172] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps described in any of the above method embodiments.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described in any of the above method embodiments.

[0174] In one embodiment, a chip is provided, including a processor for calling and running a computer program from memory, such that all devices equipped with the chip perform the steps described in any of the above method embodiments.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic quantization method, characterized in that, The method, applied to a neural network processing unit, includes: Obtain the first maximum value in the set to be quantized, where the first maximum value is the maximum absolute value of multiple data to be quantized in the set to be quantized; The quantization parameter is determined based on the first maximum value and the second maximum value, where the second maximum value is the maximum value that the target data type can represent. The data to be quantized is quantized according to the quantization parameters to obtain the quantized data corresponding to each data to be quantized.

2. The method according to claim 1, characterized in that, The step of determining the quantization parameters based on the first maximum value and the second maximum value includes: Determine a first ratio between the first maximum value and the second maximum value; The quantization parameter is determined based on the first ratio.

3. The method according to claim 2, characterized in that, The step of quantizing each of the data to be quantized according to the quantization parameters to obtain the quantized data corresponding to each of the data to be quantized includes: For each of the data to be quantized, a second ratio between the data to be quantized and the quantization parameter is determined; If the second ratio is within the target data range, the second ratio is rounded or truncated to obtain the quantized data, where the target data range is the data range that the quantized target data type can represent. If the second ratio is not within the target data range, the quantized data is determined from the boundary values ​​of the target data range.

4. The method according to claim 3, characterized in that, The data types of the data to be quantized and the first ratio are floating-point data, including sign bits, exponent bits, and mantissa bits. Determining the quantization parameters based on the first ratio includes: When all the last digits of the first ratio are 0, the exponent of the first ratio is determined as the quantization parameter; When the last digit of the first ratio is not all zero, the sum of the exponent of the first ratio and the preset value is used as the quantization parameter.

5. The method according to claim 4, characterized in that, Determining the second ratio of the data to be quantized to the quantization parameter includes: The offset value corresponding to the exponent bit is determined based on the bit width of the exponent bit of the data to be quantized. Determine the difference between the exponent of the data to be quantized and the quantization parameter, and use the sum of the difference and the offset value as the target exponent; The sign bit, the target exponent bit, and the mantissa bit of the data to be quantized are concatenated to obtain the second ratio.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain multiple data points to be quantified; The multiple data to be quantized are grouped according to a preset number of elements to obtain multiple sets of data to be quantized.

7. A dynamic quantization device, characterized in that, The device, applied to a neural network processing unit, includes: The acquisition module is used to acquire the first maximum value in the set to be quantized, wherein the first maximum value is the maximum value of the absolute values ​​of multiple data to be quantized in the set to be quantized; The determination module is used to determine the quantization parameter based on the first maximum value and the second maximum value, wherein the second maximum value is the maximum value that the target data type can represent. The quantization module is used to perform quantization processing on each of the data to be quantized according to the quantization parameters, so as to obtain the quantized data corresponding to each of the data to be quantized.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images