Data processing method and device based on neural network model, medium and equipment

Abstract

The embodiment of the invention discloses a data processing method and device based on a neural network model, a medium and equipment. The method comprises the following steps: determining original output tensor data calculated by a current operator in a neural network model; determining a data precision format of the original output tensor data, and determining a numerical range of the original output tensor data; determining a target data precision format matched with a next operator of the current operator in the neural network model, and determining a target numerical range corresponding to the target data precision format; in response to the fact that the data precision format of the original output tensor data is different from the target data precision format, determining a target quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range; and based on the target quantization parameter, performing quantization processing on the original output tensor data to obtain quantized output tensor data, and inputting the quantized output tensor data into the next operator. The calculation efficiency of the neural network model can be improved.

Description

Technical Field

[0001] The present disclosure relates to neural network model technology, and in particular to a data processing method, device, medium and equipment based on a neural network model. Background Art

[0002] During the operation of the neural network model, data quantization may be required. In the related art, fixed quantization parameters are usually used when quantizing data. The data quantized by the fixed quantization parameters is likely to be inconsistent with the data precision format required by the operator to be calculated on the data in the neural network model, which is likely to affect the computational efficiency of the neural network model. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a data processing method, device, medium and equipment based on a neural network model.

[0004] According to one aspect of an embodiment of the present disclosure, a data processing method based on a neural network model is provided, comprising:

[0005] Determine the original output tensor data calculated by the current operator in the neural network model;

[0006] Determining a data precision format of the original output tensor data, and determining a numerical range of the original output tensor data;

[0007] Determine a target data precision format adapted to a next operator of the current operator in the neural network model, and determine a target numerical range corresponding to the target data precision format;

[0008] In response to a data precision format of the original output tensor data being different from the target data precision format, determining a target quantization parameter corresponding to the current operator based on a numerical range of the original output tensor data and the target numerical range;

[0009] Based on the target quantization parameter, the original output tensor data is quantized to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

[0010] According to another aspect of the embodiments of the present disclosure, there is provided a data processing device based on a neural network model, comprising:

[0011] A first computing unit is used to determine the original output tensor data calculated by the current operator in the neural network model; determine the data precision format of the original output tensor data, and determine the value range of the original output tensor data;

[0012] The second computing unit is used to determine a target data precision format adapted to a next operator of the current operator in the neural network model, and to determine a target numerical range corresponding to the target data precision format; in response to the data precision format of the output tensor data being different from the target data precision format, based on the numerical range of the original output tensor data and the target numerical range, determine a target quantization parameter corresponding to the current operator; based on the target quantization parameter, quantize the original output tensor data to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

[0013] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned data processing method based on the neural network model.

[0014] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned data processing method based on the neural network model.

[0018] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When the instructions in the computer program product are executed by a processor, the above-mentioned data processing method based on the neural network model is performed.

[0019] Based on the data processing method, device, medium, equipment and program product based on the neural network model provided by the above embodiments of the present disclosure, the original output tensor data is the output tensor data obtained by the current operator after the input tensor data is calculated. If the data precision format of the original output tensor data is different from the target data precision format, that is, the data precision format of the original output tensor data does not meet the data precision format requirements of the next operator, then the target quantization parameter corresponding to the current operator can be determined based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format, so as to quantize the original output tensor data, thereby obtaining quantized output tensor data that meets the data precision format requirements of the next operator. It can be seen that in the embodiments of the present disclosure, when the data is quantized during the operation of the neural network model, a fixed quantization parameter is not used, but the precision of the output tensor data after the current operator is dynamically adjusted based on the data precision format requirements of the next operator, so that the precision format of the adjusted output tensor data is adapted to the data precision format requirements of the next operator, which is conducive to improving the computational efficiency of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is one of the flowcharts of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0021] Figure 2 It is a schematic diagram of the structure of a neural network model in some exemplary embodiments of the present disclosure.

[0022] Figure 3 This is the second flowchart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0023] Figure 4-1 This is the third flow chart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0024] Figure 4-2 This is the fourth flowchart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0025] Figure 5 This is the fifth flowchart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0026] Figure 6 This is the sixth flow chart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0027] Figure 7This is the seventh flow chart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0028] Figure 8 This is one of the schematic diagrams of some exemplary data processing methods based on neural network models in the present disclosure.

[0029] Fig. 9 This is the eighth flow chart of the data processing method based on the neural network model provided by some exemplary embodiments of the present disclosure.

[0030] Figure 10-1 This is the second schematic diagram of some exemplary data processing methods based on neural network models in the present disclosure.

[0031] Figure 10-2 This is the third schematic diagram of some exemplary data processing methods based on neural network models in the present disclosure.

[0032] Figure 10-3 This is the fourth schematic diagram of some exemplary data processing methods based on neural network models in the present disclosure.

[0033] Fig.11 It is one of the structural schematic diagrams of a data processing device based on a neural network model provided by some exemplary embodiments of the present disclosure.

[0034] Fig.12 This is the second structural schematic diagram of a data processing device based on a neural network model provided by some exemplary embodiments of the present disclosure.

[0035] Fig.13 It is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0037] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0038] Application Overview

[0039] Neural network models are widely used in many fields such as computer vision, natural language processing, speech recognition, and autonomous driving. For example, using neural network models, target detection, target tracking, semantic segmentation, and instance segmentation can be performed on environmental images collected by vehicle cameras.

[0040] It is understandable that the neural network model may include multiple operators arranged in a certain order. The multiple operators may include but are not limited to convolution operators, pooling operators, deconvolution operators, rectified linear unit (Rectified Linear Unit, ReLU) operators, point-to-point element-wise operators, normalization operators, etc.

[0041] Because different operators may have different requirements for data precision formats, data quantization may be required during the operation of the neural network model. For example, after an operator in the neural network model obtains the first feature data through calculation, it is necessary to first quantize the first feature data to obtain the second feature data, and then input the second feature data into the next operator for calculation. In the related art, when quantizing data, fixed quantization parameters are usually used. The data quantized by the fixed quantization parameters is likely to be inconsistent with the data precision format required by the next operator (i.e., the operator to be calculated on the data), which can easily affect the computational efficiency of the neural network model.

[0042] Exemplary Systems

[0043] The neural network model can be run on a neural network processing unit (NPU), which can include a tensor computing unit, a vector computing unit, and a cache.

[0044] A tensor computing unit may be a hardware unit for performing tensor computing. Tensor computing may also be referred to as matrix computing. As an example, tensor computing may include convolution computing. Convolution computing may include, but is not limited to, two-dimensional convolution computing, three-dimensional convolution computing, and the like.

[0045] The vector calculation unit may be a hardware unit for performing vector calculations. Vector calculations may also be referred to as vector calculations. As an example, vector calculations may include point-to-point element-by-element calculations. Point-to-point element-by-element calculations may include, but are not limited to, point-to-point element-by-element multiplication calculations, point-to-point element-by-element addition calculations, and the like.

[0046] In addition, quantization processing is the process of converting data between different data precision formats. The data precision format can be understood as the representation accuracy of the data. The data precision format can include a fixed-point data precision format and a floating-point data precision format. As an example, the fixed-point data precision format can include INT8, INT16, etc. The floating-point data precision format can include FP8, FP16, FP32, etc.

[0047] Exemplary Methods

[0048] Figure 1 It is a flowchart of a data processing method based on a neural network model provided by some exemplary embodiments of the present disclosure. Figure 1 The method shown may include step 110 , step 120 , step 130 , step 140 , and step 150 .

[0049] Step 110, determining the original output tensor data calculated by the current operator in the neural network model.

[0050] Optionally, the neural network model can be used to implement a specific task. The specific task may include, but is not limited to, a target detection task, a target tracking task, a semantic segmentation task, an instance segmentation task, etc. The neural network model may include multiple operators arranged in a certain order, and each operator except the operator at the end of the multiple operators may be used as the current operator.

[0051] In one example, the neural network model can be a Transformer model; wherein the Transformer model is a model based on an encoder-decoder architecture and an attention mechanism. The structure of the Transformer model can be referred to Figure 2 , Figure 2 Each rectangular box in can be used to represent an operator. Figure 2 Every operator except the rightmost operator can be used as the current operator.

[0052] Optionally, if the current operator is the first operator in the neural network model, the input tensor data of the current operator may be image information to be executed for a specific task. As an example, the image information may be a single image or a sequence of images. If the current operator is not the first operator in the neural network model, the input tensor data of the current operator may be a feature map output by the previous operator of the current operator in the neural network model. The current operator may calculate the input tensor data to obtain another feature map, which may be used as the original output tensor data.

[0053] Step 120, determining the data precision format of the original output tensor data, and determining the numerical range of the original output tensor data.

[0054] Optionally, the data precision format of the original output tensor data can be understood as the representation precision of the original output tensor data.

[0055] Optionally, the original output tensor data is multidimensional data, so the original output tensor data includes data of multiple pixels in the image. The original output tensor data may include multiple values, and the value range of the original output tensor data may be understood as a range that at least covers all values ​​in the original output tensor data. The value range of the original output tensor data may be a range with an upper limit value and a lower limit value. In an example, the lower limit value may be expressed as d1, the upper limit value may be expressed as d2, and the value range of the original output tensor data may be expressed as [d1, d2].

[0056] Step 130, determining a target data precision format adapted to the next operator of the current operator in the neural network model, and determining a target numerical range corresponding to the target data precision format.

[0057] Generally speaking, each operator in a neural network model has certain requirements for the data precision format of the tensor data input to it. Only when the tensor data meets the requirements can the operator perform calculations normally and efficiently for the tensor data. The target data precision format adapted to the next operator of the current operator in the neural network model can be understood as a data precision format that meets the data precision format requirements of the next operator. The target numerical range corresponding to the target data precision format can be understood as the range of values ​​of numbers that can be represented under the target data precision format. For example, if the target data precision format is INT8, the target numerical range corresponding to the target data precision format can be [-128, 127].

[0058] Step 140 , in response to the data precision format of the original output tensor data being different from the target data precision format, determine a target quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range.

[0059] Optionally, it can be determined whether the data precision format of the original output tensor data is the same as the target data precision format. If the data precision format of the original output tensor data is different from the target data precision format, this means that the data precision format of the original output tensor data does not meet the data precision format requirements of the next operator. Then, the target quantization parameter corresponding to the current operator can be determined based on the numerical range of the original output tensor data and the target numerical range. The target quantization parameter corresponding to the current operator may be a quantization parameter used to transform the numerical range of the original output tensor data to the target numerical range. If the quantization processing involved in the embodiment of the present disclosure is a symmetric quantization processing, the target quantization parameter corresponding to the current operator may include only a scaling factor. If the quantization processing involved in the embodiment of the present disclosure is an asymmetric quantization processing, the target quantization parameter corresponding to the current operator may include a scaling factor and a zero point.

[0060] It can be understood that symmetric quantization and asymmetric quantization are two conventional quantization methods. In the quantization method of symmetric quantization, the zero points before and after quantization remain consistent, while in the quantization method of asymmetric quantization, the zero points before and after quantization are inconsistent.

[0061] Step 150 , based on the target quantization parameter, quantize the original output tensor data to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

[0062] Optionally, the target quantization parameter may include a scaling factor and a zero point, and the scaling factor may be expressed as X scale , the zero point can be represented as X zero point Any value in the original output tensor data can be represented as X 原始 , the quantized output tensor data is the same as X 原始 The corresponding value can be expressed as X 量化 , then X 原始 Can be equal to X scale The product of the target difference; where the target difference is X 量化 With X zero point That is, X 原始 =X scale *(X 量化 -X zero point ). Then, by substituting each value in the original output tensor data, as well as the scaling factor and the zero point into the above formula for calculation, the quantized output tensor data can be obtained.

[0063] In some embodiments, the target quantization parameter may include only the scaling factor, X zero point It can be set to zero by default, then X 原始 Can be equal to X scale With X 量化 That is, X 原始 =X scale *X 量化 Then, by substituting each value in the original output tensor data and the scaling factor into the above formula for calculation, the quantized output tensor data can be obtained.

[0064] After obtaining the quantized output tensor data, the quantized output tensor data can be input into the next operator. The quantized output tensor data can be used as the input tensor data of the next operator. If the next operator is not the operator at the end of the neural network model, the next operator can be regarded as a new current operator, and the above steps 110 to 150 are performed again for the new current operator.

[0065] In the embodiments of the present disclosure, the original output tensor data is the output tensor data obtained by calculating the input tensor data by the current operator. If the data precision format of the original output tensor data is different from the target data precision format, that is, the data precision format of the original output tensor data does not meet the data precision format requirements of the next operator, then the target quantization parameter corresponding to the current operator can be determined based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format, so as to quantize the original output tensor data, thereby obtaining quantized output tensor data that meets the data precision format requirements of the next operator. It can be seen that in the embodiments of the present disclosure, when quantizing data during the operation of the neural network model, fixed quantization parameters are not used, but the precision of the output tensor data calculated by the current operator is dynamically adjusted based on the data precision format requirements of the next operator, so that the data precision format of the adjusted output tensor data is adapted to the data precision format requirements of the next operator, which is conducive to improving the computational efficiency of the neural network model.

[0066] In some optional examples, such as Figure 3 As shown, determining the numerical range of the original output tensor data in step 120 may include steps 310 and 320 .

[0067] Step 310, determining the maximum value and the minimum value of all the values ​​included in the original output tensor data.

[0068] Optionally, statistics may be performed on all values ​​included in the original output tensor data to obtain statistical information of the original output tensor data. The statistical information may include a maximum value and a minimum value among all values ​​included in the original output tensor data.

[0069] Step 320, determining the numerical range of the original output tensor data based on the maximum value and the minimum value.

[0070] Optionally, the upper limit value may be determined to be the maximum value determined in step 310, and the lower limit value may be determined to be a numerical range of the minimum value determined in step 310, and the numerical range may be used as the numerical range of the original output tensor data. In some embodiments, the upper limit value of the numerical range of the original output tensor data may be slightly greater than the maximum value determined in step 310, and / or the lower limit value of the numerical range of the original output tensor data may be slightly less than the minimum value determined in step 310.

[0071] In one example, the original output tensor data is a feature map with a height of H, a width of W, and a number of channels of C, and the original output tensor data may include H*W*C values. Then, the maximum value of the H*W*C values ​​may be used as d2 in the above text, and the minimum value of the H*W*C values ​​may be used as d1 in the above text, thereby obtaining [d1, d2], that is, obtaining the value range of the original output tensor data.

[0072] In the embodiments of the present disclosure, by performing statistics on all values ​​included in the original output tensor data, the value range of the original output tensor data can be obtained efficiently and reliably.

[0073] In some optional examples, such as Figure 4-1 As shown, determining the target data precision format adapted to the next operator of the current operator in the neural network model in step 130 may include steps 410, 420, 430, and 440.

[0074] Step 410: Determine the operator type of the next operator.

[0075] Optionally, the operator type of the next operator can be used at least to characterize the calculation type corresponding to the next operator. For example, if the next operator is a convolution operator, the operator type of the next operator can be used to characterize that the calculation type corresponding to the next operator is a convolution calculation. For another example, if the next operator is a pooling operator, the operator type of the next operator can be used to characterize that the calculation type corresponding to the next operator is a pooling type. For another example, if the next operator is a point-to-point element-by-element operator, the operator type of the next operator can be used to characterize that the calculation type corresponding to the next operator is a point-to-point element-by-element calculation.

[0076] Step 420, determine the hardware unit in the neural network accelerator that corresponds to the operator type of the next operator.

[0077] Optionally, the neural network accelerator may include a tensor computing unit and a vector computing unit, and then a hardware unit corresponding to the operator type of the next operator may be determined from the tensor computing unit and the vector computing unit.

[0078] In an example, the next operator is a convolution operator, and the operator type of the next operator is used to characterize that the calculation type corresponding to the next operator is convolution calculation. Since convolution calculation belongs to tensor calculation, the tensor calculation unit can be determined as a hardware unit corresponding to the operator type of the next operator.

[0079] In another example, the next operator is a point-to-point element-by-element operator, and the operator type of the next operator is used to characterize that the calculation type corresponding to the next operator is point-to-point element-by-element calculation. Since the point-to-point element-by-element calculation belongs to vector calculation, the vector calculation unit can be determined as a hardware unit corresponding to the operator type of the next operator.

[0080] Step 430: determine the data precision format supported by the hardware unit corresponding to the operator type of the next operator.

[0081] Optionally, the data precision formats supported by the tensor computing unit may include FP8 and FP16; the data precision formats supported by the vector computing unit may include FP16. In this way, if the hardware unit corresponding to the operator type of the next operator is a tensor computing unit, the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator may be multiple, namely FP8 and FP16. If the hardware unit corresponding to the operator type of the next operator is a vector computing unit, the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator may be one, namely FP16.

[0082] Step 440 , based on the data precision format supported by the hardware unit corresponding to the operator type of the next operator, determine the target data precision format adapted to the next operator.

[0083] Optionally, if the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator is one, this data precision format can be used as the target data precision format adapted to the next operator. If the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator is multiple, one data precision format can be selected from the multiple data precision formats supported by the hardware unit corresponding to the operator type of the next operator as the target data precision format.

[0084] In an embodiment of the present disclosure, in the process of determining the target data precision format, information related to the hardware unit corresponding to the operator type of the next operator (i.e., the supported data precision format) can be used. Since the quantized output tensor data of the current operator needs to be input to the next operator for calculation, and the calculation needs to rely on the hardware unit corresponding to the operator type of the next operator, this is conducive to ensuring the normal and efficient subsequent calculation of the quantized output tensor data.

[0085] In some optional examples, such as Figure 4-2 As shown, step 440 may include step 4401 and step 4403.

[0086] Step 4401 , in response to a hardware unit corresponding to an operator type of a next operator supporting multiple data precision formats, determine a bit width of each data precision format.

[0087] Optionally, if the hardware unit corresponding to the operator type of the next operator is a tensor computing unit, the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator may be multiple, for example, including FP8 and FP16. Then, the bit widths of FP8 and FP16 can be determined. Obviously, the bit width of FP8 is 8, and the bit width of FP16 is 16.

[0088] Step 4403 , based on the bit width of each data precision format, determine a target data precision format adapted to the next operator from a plurality of data precision formats supported by a hardware unit corresponding to the operator type of the next operator.

[0089] Optionally, a data precision format with the smallest bit width can be selected as the target data precision format from a plurality of data precision formats supported by a hardware unit corresponding to the operator type of the next operator, so as to reduce the bandwidth required for data memory access and improve performance.

[0090] In some embodiments, the hardware unit corresponding to the operator type of the next operator supports a large number of data precision formats, for example, five, and the data precision format with the second smallest or third smallest bit width may be selected as the target data precision format.

[0091] In an embodiment of the present disclosure, when the number of data precision formats supported by the hardware unit corresponding to the operator type of the next operator is multiple, the bit widths of the multiple data precision formats can be used as a selection basis to select a target data precision format from the multiple data precision formats, which is conducive to ensuring the rationality of the selected target data precision format.

[0092] In some optional examples, such as Figure 5 As shown, the method for determining the target quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range in step 140 may include steps 510 and 520.

[0093] Step 510 , determining a scaling factor for transforming the value range of the original output tensor data to the target value range.

[0094] Optionally, the numerical range of the original output tensor data can be expressed as [d1, d2], and the target numerical range can be expressed as [d3, d4]. Then, the first difference between d2 and d1 can be calculated, the second difference between d4 and d3 can be calculated, and the ratio of the first difference to the second difference can be calculated. The ratio can be used as a scaling factor for transforming the numerical range of the original output tensor data to the target numerical range. That is, X scale =(d2-d1) / (d4-d3).

[0095] Step 520: Determine a target quantization parameter corresponding to the current operator based on the scaling factor.

[0096] Alternatively, the target quantization parameter may include only the scaling factor.

[0097] In some embodiments, the target quantization parameter may include a scaling factor and a zero point. The following formula is introduced above: 原始 =X scale *(X 量化 -X zero point ), by transforming this formula, we can get the following formula: X zero point =X 量化 -X 原始 / X scale The zero point in the target quantization parameter can be calculated using the transformed formula.

[0098] In the embodiments of the present disclosure, the target numerical range can be understood as the numerical range to which quantization is required, and the numerical range of the original output tensor data can be understood as the numerical range before quantization. By combining the numerical range to which quantization is required and the numerical range before quantization, the scaling factor can be reasonably determined, and on this basis, the target quantization parameter can be determined, which is conducive to ensuring the rationality of the determined target quantization parameter.

[0099] In some optional examples, such as Figure 6 As shown, determining the target quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range in step 140 may include steps 610 , 620 , and 630 .

[0100] Step 610 , determining a reference quantization parameter corresponding to the current operator based on the value range of the original output tensor data and the target value range.

[0101] Optionally, the formula X in the above text can be used. scale =(d2-d1) / (d4-d3) to determine the scaling factor to obtain a reference quantization parameter including the determined scaling factor.

[0102] In some embodiments, the reference quantization parameter may further include a zero point. The zero point in the reference quantization parameter may be obtained using the following formula in the above text: zero point =X 量化 -X 原始 / X scale .

[0103] Step 620 , determining a historical quantization parameter for quantizing the historical output tensor data calculated by the current operator.

[0104] Specifically, the original output tensor data is the output tensor data currently calculated by the current operator, and the historical output tensor data is the output tensor data obtained after the current operator calculates the previous input tensor data, or the output tensor data obtained after the current operator calculates the previous multiple input tensor data. The method for determining the historical quantization parameter refers to the method for determining the reference quantization parameter corresponding to the current operator, and the similarities are not repeated here.

[0105] Step 630 : Determine a target quantization parameter corresponding to the current operator based on the reference quantization parameter and the historical quantization parameter.

[0106] Optionally, a quantization parameter may be reasonably selected from the reference quantization parameter and the historical quantization parameter as the target quantization parameter corresponding to the current operator. Alternatively, the reference quantization parameter and the historical quantization parameter may be fused to obtain the target quantization parameter corresponding to the current operator.

[0107] In some optional embodiments of the present disclosure, Figure 7 As shown, step 630 may include step 710 , step 720 , step 730 , step 740 and step 750 .

[0108] Step 710 , in response to the number of historical quantization parameters being multiple, selecting historical quantization parameters that meet a preset numerical condition from the multiple historical quantization parameters.

[0109] Optionally, the multiple historical quantization parameters may be represented as N historical quantization parameters, where N may be an integer greater than or equal to 2. Each of the N historical quantization parameters may include only one scaling factor, and thus the N historical quantization parameters may be N scaling factors. The historical quantization parameter that satisfies the preset numerical condition may be the scaling factor with the largest value among the N scaling factors.

[0110] In some embodiments, if the value of N is very large, the local maximum value among the N scaling factors can be used as the historical quantization parameter that meets the preset numerical condition. For example, N can be 50, then the 10 scaling factors closest to the current calculation time can be determined from the 50 scaling factors, and the 10 scaling factors closest to the current calculation time can correspond to Figure 8 In the current statistical window, the historical quantitative parameter that meets the preset numerical condition can be the scaling factor with the largest value in the current statistical window.

[0111] Step 720: Determine a difference evaluation value between a historical quantization parameter that meets a preset numerical condition and a reference quantization parameter.

[0112] Optionally, a difference between a historical quantization parameter satisfying a preset numerical condition and a reference quantization parameter may be calculated, and the calculated difference may be used as a difference evaluation value.

[0113] It should be noted that the difference involved in the above paragraph is only explained by taking the arithmetic difference as an example. In specific implementation, the standard deviation, variance, etc. can also be calculated for the historical quantitative parameters and reference quantitative parameters that meet the preset numerical conditions, and the calculated standard deviation, variance, etc. can be used as the difference evaluation value.

[0114] Step 730: Determine distribution information of the difference evaluation value relative to a preset evaluation value range.

[0115] Optionally, the distribution information of the difference evaluation value relative to the preset evaluation value range may be used to indicate whether the difference evaluation value is within the preset evaluation value range.

[0116] Step 740 , selecting a quantization parameter that matches the distribution information from a plurality of historical quantization parameters and reference quantization parameters.

[0117] Optionally, if the distribution information is used to characterize that the difference evaluation value is within a preset evaluation value range, at least some of the N historical quantization parameters may be determined as quantization parameters adapted to the distribution information. For example, the historical quantization parameter most recently used for actual quantization processing among the N historical quantization parameters (hereinafter referred to as the most recently used quantization parameter for ease of description) may be determined as the quantization parameter adapted to the distribution information.

[0118] Optionally, if the distribution information is used to characterize that the difference evaluation value is outside the preset evaluation value range, only the reference quantization parameter may be determined as the quantization parameter adapted to the distribution information, or the reference quantization parameter and at least part of the N historical quantization parameters may be determined as the quantization parameters adapted to the distribution information. For example, the reference quantization parameter and the most recently used quantization parameter may be determined as the quantization parameters adapted to the distribution information.

[0119] Step 750: Determine a target quantization parameter corresponding to the current operator based on the quantization parameter adapted to the distribution information.

[0120] Optionally, if the most recently used quantization parameter is determined as the quantization parameter adapted to the distribution information in step 740 , the most recently used quantization parameter may be determined as the target quantization parameter.

[0121] Optionally, if the reference quantization parameter is determined as a quantization parameter adapted to the distribution information in step 740 , the reference quantization parameter may be determined as the target quantization parameter.

[0122] Optionally, if the reference quantization parameter and the most recently used quantization parameter are determined as quantization parameters adapted to the distribution information in step 740, a mean operation or a weighted average operation can be performed on the reference quantization parameter and the most recently used quantization parameter, and the obtained mean operation result or weighted average operation result can be determined as the target quantization parameter.

[0123] In this way, when there are multiple historical quantization parameters, the difference evaluation value between the historical quantization parameters and the reference quantization parameters that meet the preset numerical conditions can be determined. According to the distribution information of the difference evaluation value relative to the preset evaluation value range, the quantization parameters can be reasonably screened, and the target quantization parameter can be determined on this basis. That is, the basis for determining the target quantization parameter includes not only the reference quantization parameter, but also multiple historical quantization parameters. The basis for determining the target quantization parameter is very rich, which is conducive to ensuring the rationality of the determined target quantization parameter. For example, using multiple historical quantization parameters to determine the target quantization parameter is conducive to ensuring the smoothness and stability of the quantization parameter used for the quantization operation of the output tensor data of the current operator, and avoiding the adverse effects caused by excessive fluctuations in the quantization parameter.

[0124] In some embodiments, the number of historical quantization parameters may be only one, and a difference evaluation value between the historical quantization parameter and the reference quantization parameter may be determined. Figure 7 The illustrated embodiments are similar and will not be described in detail here.

[0125] In the embodiments of the present disclosure, the target numerical range can be understood as the numerical range to be quantized, and the numerical range of the original output tensor data can be understood as the numerical range before quantization. By combining the numerical range to be quantized and the numerical range before quantization, the reference quantization reference can be reasonably determined. By combining the historical quantization parameters, the target quantization parameters can be adaptively determined, which is conducive to ensuring the rationality of the determined target quantization parameters and the operating effect of the neural network model.

[0126] In some optional examples, such as Fig. 9 As shown, after step 130 , the method provided by the embodiment of the present disclosure may further include step 910 .

[0127] Step 910 , in response to the data precision format of the original output tensor data being the same as the target data precision format, inputting the original output tensor data into the next operator.

[0128] Optionally, it can be determined whether the data precision format of the original output tensor data is the same as the target data precision format. If the data precision format of the original output tensor data is the same as the target data precision format, it means that the data precision format of the original output tensor data meets the data precision format requirement of the next operator. Then, the original output tensor data can be directly input into the next operator without quantization processing, and the original output tensor data can be correctly calculated.

[0129] In the embodiments of the present disclosure, it can be determined whether quantization processing is required based on the consistency between the data precision format of the original output tensor data and the target data precision format. If the data precision format of the original output tensor data is different from the target data precision format, it can be determined that quantization processing is required, then the quantization processing operation on the original output tensor data can be performed to obtain the quantized output tensor data and provide it to the next operator for calculation. If the data precision format of the original output tensor data is the same as the target data precision format, it can be determined that quantization processing is not required, then the original output tensor data can be directly provided to the next operator for calculation. In this way, it is beneficial to ensure the correct calculation of the data and save resources and power consumption.

[0130] In some optional examples, such as Figure 10-1 As shown, for the current operator, a statistics unit 1010, an analysis unit 1020 and an adaptive quantization unit 1030 may be set. The statistics unit 1010 may run in a tensor calculation unit of a neural network accelerator. The analysis unit 1020 and the adaptive quantization unit 1030 may run in a vector calculation unit of a neural network accelerator.

[0131] The current operator can obtain the input tensor data from the previous operator, and calculate the input tensor data to obtain the original output tensor data. The statistical unit 1010 can perform data statistics on the original output tensor data to obtain statistical information. The statistical information may include the maximum and minimum values ​​in the original output tensor data. In this way, the statistical unit 1010 can determine the numerical range of the original output tensor data, and the statistical unit 1010 can transmit the numerical range of the original output tensor data to the analysis unit 1020. Accordingly, the analysis unit 1020 can obtain the numerical range of the original output tensor data. In addition, the analysis unit 1020 can also obtain the operator type of the next operator. Based on the operator type of the next operator, the analysis unit 1020 can determine the target data precision format adapted to the next operator (for example, it can be based on Figure 4-1Based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format, the analysis unit 1020 may determine a reference quantization parameter (for example, the reference quantization parameter may be determined in the manner described above in relation to step 610).

[0132] If the data precision format of the original output tensor data is the same as the target data precision format, the original output tensor data can be directly input into the next operator (for example, see Fig. 9 If the data precision format of the original output tensor data is different from the target data precision format, the adaptive quantization unit 1030 may adaptively determine the target quantization parameter based on the reference quantization parameter and the historical quantization parameter (for example, according to Figure 7 In the embodiment shown, the target quantization parameter is determined. Specifically, the degree of difference between the reference quantization parameter and the historical quantization parameter can be determined. If the difference between the reference quantization parameter and the historical quantization parameter is greater than a preset value, the weighted average result of the reference quantization parameter and the historical quantization parameter can be used as the target quantization parameter. If the difference between the reference quantization parameter and the historical quantization parameter is less than or equal to the preset value, the historical quantization parameter can be used as the target quantization parameter. The target quantization parameter can be used for quantization processing of the original output tensor data, thereby obtaining quantized output tensor data, so that the quantized output tensor data can be input into the next operator.

[0133] In some optional examples, the current operator can obtain the input tensor data from the previous operator and perform calculations on the input tensor data to obtain the original output tensor data. Figure 10-2 As shown, the original output tensor data can be high-bit precision data, for example, the data precision format of the original output tensor data can be FP32. Next, data statistics can be performed on the original output tensor data to obtain the maximum and minimum values ​​in the original output tensor data, thereby determining the value range of the original output tensor data. In addition, the target data precision format can be determined based on the operator type of the next operator (for example, according to Figure 4-1 The target data precision format is determined in the manner of the embodiment shown in the figure). Based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format, the target quantization parameter can be determined (for example, according to Figure 5 The target quantization parameter is determined by the method in the embodiment shown in the figure). The target quantization parameter can be used for quantization processing of the original output tensor data, thereby obtaining quantized output tensor data, so as to input the quantized output tensor data into the next operator. The quantized output tensor data can be low-bit precision data, for example, the data precision format of the quantized output tensor data can be FP16 or FP8.

[0134] It should be noted that Figure 10-3 The specific implementation of the embodiment shown is Figure 10-2 The specific implementation of the illustrated embodiment may be similar, the difference mainly being that: Figure 10-3 In the embodiment shown, based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format, the target quantization parameter can be determined by referring to the historical quantization parameter (for example, see Figure 7 Implementation shown).

[0135] In some optional examples, the current operator can obtain the input tensor data from the previous operator and perform calculations on the input tensor data to obtain the original output tensor data. Figure 8 As shown, the original output tensor data can be high-bit precision data, for example, the data precision format of the original output tensor data can be FP32. Next, data statistics can be performed on the original output tensor data to obtain the maximum and minimum values ​​in the original output tensor data, thereby determining the value range of the original output tensor data. In addition, the target data precision format can be determined based on the operator type of the next operator (for example, according to Figure 4-1 The target data precision format is determined in the manner of the embodiment shown in the figure). The target quantization parameter can be determined based on the numerical range of the original output tensor data and the target numerical range corresponding to the target data precision format. For example, the reference quantization parameter can be determined based on the numerical range of the original output tensor data and the target numerical range, and the historical quantization parameter that meets the preset numerical condition can be selected from multiple historical quantization parameters. The historical quantization parameter that meets the preset numerical condition can be Figure 8 The maximum scaling factor in the current statistical window shown in FIG. 4 is then determined based on the historical quantization parameters and reference quantization parameters that meet the preset numerical conditions (for example, the target quantization parameter can be determined using Figure 7 The target quantization parameter is determined in the embodiment shown in the figure). The target quantization parameter can be used for quantization processing of the original output tensor data, so that quantized output tensor data can be obtained, so that the quantized output tensor data can be input into the next operator. The quantized output tensor data can be low-bit precision data, for example, the data precision format of the quantized output tensor data can be FP16 or FP8.

[0136] In summary, by using the embodiments of the present disclosure, when performing data quantization processing during the operation of the neural network model, fixed quantization parameters are not used, but the quantization parameters are determined adaptively. This can dynamically manage operator precision during operation, which is beneficial to ensuring the accuracy and computational efficiency of the neural network model, and can achieve better performance with lower memory utilization.

[0137] Exemplary Devices

[0138] Fig.11 It is a structural diagram of a data processing device based on a neural network model provided by some exemplary embodiments of the present disclosure. Fig.11 The apparatus shown may include:

[0139] The first calculation unit 1110 is used to determine the original output tensor data calculated by the current operator in the neural network model; determine the data precision format of the original output tensor data, and determine the value range of the original output tensor data;

[0140] The second computing unit 1120 is used to determine a target data precision format that is compatible with the next operator of the current operator in the neural network model, and to determine a target numerical range corresponding to the target data precision format; in response to the data precision format of the output tensor data being different from the target data precision format, based on the numerical range of the original output tensor data and the target numerical range, determine a target quantization parameter corresponding to the current operator; based on the target quantization parameter, quantize the original output tensor data to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

[0141] Optionally, the first computing unit 1110 may be a tensor computing unit, and the second computing unit 1120 may be a vector computing unit. Of course, the types of the first computing unit 1110 and the second computing unit 1120 are not limited thereto. For example, the first computing unit 1110 may be a central processing unit (CPU), and the second computing unit 1120 may be a graphics processing unit (GPU). It is sufficient to ensure that the first computing unit 1110 and the second computing unit 1120 can perform the required corresponding calculations, and the present disclosure does not limit this.

[0142] In some optional examples, the second computing unit 1120 may include Fig.12 The analysis unit 1020 shown; wherein,

[0143] The analysis unit 1020 is used to determine the operator type of the next operator; determine the hardware unit corresponding to the operator type of the next operator in the neural network accelerator; determine the data precision format supported by the hardware unit; and determine the target data precision format adapted to the next operator based on the data precision format supported by the hardware unit.

[0144] In some optional examples, the analysis unit 1020 is used to determine the bit width of each data precision format in response to the hardware unit supporting multiple data precision formats; based on the bit width of each data precision format, determine the target data precision format adapted to the next operator from the multiple data precision formats supported by the hardware unit.

[0145] In some optional examples, the second computing unit 1120 may include Fig.12 The adaptive quantization unit 1030 shown; wherein,

[0146] The adaptive quantization unit 1030 is used to determine a scaling factor for transforming the numerical range of the original output tensor data to the target numerical range; based on the scaling factor, determine a target quantization parameter corresponding to the current operator.

[0147] In some optional examples, the second computing unit 1120 may include Fig.12 The adaptive quantization unit 1030 shown; wherein,

[0148] The adaptive quantization unit 1030 is used to determine a reference quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range; determine a historical quantization parameter used to quantize the historical output tensor data calculated by the current operator; and determine a target quantization parameter corresponding to the current operator based on the reference quantization parameter and the historical quantization parameter.

[0149] In some optional examples, the adaptive quantization unit 1030 is used to, in response to the number of historical quantization parameters being multiple, screen historical quantization parameters that meet preset numerical conditions from multiple historical quantization parameters; determine a difference evaluation value between the historical quantization parameters that meet the preset numerical conditions and a reference quantization parameter; determine distribution information of the difference evaluation value relative to a preset evaluation value range; screen quantization parameters that are adapted to the distribution information from multiple historical quantization parameters and reference quantization parameters; and determine a target quantization parameter corresponding to the current operator based on the quantization parameter that is adapted to the distribution information.

[0150] In some optional examples, the second computing unit 1120 is further used to input the original output tensor data into the next operator in response to the data precision format of the original output tensor data being the same as the target data precision format.

[0151] In some optional examples, the first computing unit 1110 may include Fig.12 The statistical unit 1010 shown; wherein,

[0152] The statistical unit 1010 is used to determine the maximum value and the minimum value of all the values ​​included in the original output tensor data; based on the maximum value and the minimum value, determine the value range of the original output tensor data.

[0153] Optionally, the statistics unit 1010 may be electrically connected to the analysis unit 1020 , and the analysis unit 1020 may be electrically connected to the adaptive quantization unit 1030 .

[0154] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0155] Exemplary Electronic Devices

[0156] Fig.13 The block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 1300 includes one or more processors 1310 and a memory 1320 .

[0157] The processor 1310 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1300 to perform desired functions.

[0158] The memory 1320 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1310 may execute one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0159] In one example, the electronic device 1300 may further include: an input device 1330 and an output device 1340 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0160] The input device 1330 may also include, for example, a keyboard, a mouse, etc.

[0161] The output device 1340 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0162] Of course, to simplify, Fig.13 Only some of the components related to the present disclosure in the electronic device 1300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1300 may also include any other appropriate components.

[0163] Exemplary computer program products and computer-readable storage media

[0164] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0165] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0166] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0167] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0168] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. The specific details disclosed above are only for the purpose of illustration and ease of understanding, not limitation. The above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0169] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A data processing method based on a neural network model, comprising: Determine the original output tensor data calculated by the current operator in the neural network model; Determining a data precision format of the original output tensor data, and determining a numerical range of the original output tensor data; Determine a target data precision format adapted to a next operator of the current operator in the neural network model, and determine a target numerical range corresponding to the target data precision format; In response to the data precision format of the output tensor data being different from the target data precision format, determining a target quantization parameter corresponding to the current operator based on a numerical range of the original output tensor data and the target numerical range; Based on the target quantization parameter, the original output tensor data is quantized to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

2. The method according to claim 1, wherein: The determining of a target data precision format adapted to a next operator of a current operator in the neural network model comprises: determining an operator type of the next operator; Determining a hardware unit in the neural network accelerator corresponding to the operator type of the next operator; Determine the data precision format supported by the hardware unit; Based on the data precision format supported by the hardware unit, a target data precision format adapted to the next operator is determined.

3. The method according to claim 2, wherein: The determining, based on the data precision format supported by the hardware unit, a target data precision format adapted to the next operator comprises: In response to the hardware unit supporting multiple data precision formats, determining a bit width of each of the data precision formats; Based on the bit width of each of the data precision formats, a target data precision format adapted to the next operator is determined from the multiple data precision formats supported by the hardware unit.

4. The method according to claim 1, wherein: The determining, based on the numerical range of the original output tensor data and the target numerical range, a target quantization parameter corresponding to the current operator includes: Determining a scaling factor for transforming the value range of the original output tensor data to the target value range; Based on the scaling factor, a target quantization parameter corresponding to the current operator is determined.

5. The method according to claim 1, wherein: The determining, based on the numerical range of the original output tensor data and the target numerical range, a target quantization parameter corresponding to the current operator includes: Determining a reference quantization parameter corresponding to the current operator based on a numerical range of the original output tensor data and the target numerical range; Determining a historical quantization parameter for quantizing historical output tensor data calculated by the current operator; Based on the reference quantization parameter and the historical quantization parameter, a target quantization parameter corresponding to the current operator is determined.

6. The method according to claim 5, wherein: The determining, based on the reference quantization parameter and the historical quantization parameter, a target quantization parameter corresponding to the current operator includes: In response to the number of the historical quantization parameters being multiple, selecting the historical quantization parameters that meet a preset numerical condition from the multiple historical quantization parameters; Determine a difference evaluation value between the historical quantization parameter and the reference quantization parameter that meets a preset numerical condition; Determining distribution information of the difference evaluation value relative to a preset evaluation value range; Selecting a quantization parameter that matches the distribution information from the plurality of historical quantization parameters and the reference quantization parameters; Based on the quantization parameter adapted to the distribution information, a target quantization parameter corresponding to the current operator is determined.

7. The method according to any one of claims 1 to 6, further comprising: In response to the data precision format of the original output tensor data being the same as the target data precision format, the original output tensor data is input to the next operator.

8. The method according to any one of claims 1 to 6, wherein: Determining the numerical range of the original output tensor data includes: Determine the maximum and minimum values ​​of all values ​​included in the original output tensor data; Based on the maximum value and the minimum value, a numerical range of the original output tensor data is determined.

9. A data processing device based on a neural network model, comprising: A first computing unit, used to determine the original output tensor data calculated by the current operator in the neural network model; Determining a data precision format of the original output tensor data, and determining a numerical range of the original output tensor data; A second computing unit is configured to determine a target data precision format adapted to a next operator of the current operator in the neural network model, and to determine a target numerical range corresponding to the target data precision format; in response to the data precision format of the output tensor data being different from the target data precision format, determine a target quantization parameter corresponding to the current operator based on the numerical range of the original output tensor data and the target numerical range; Based on the target quantization parameter, the original output tensor data is quantized to obtain quantized output tensor data, so as to input the quantized output tensor data into the next operator.

10. A computer-readable storage medium storing a computer program for executing the data processing method based on a neural network model as described in any one of claims 1 to 8.

11. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the data processing method based on the neural network model as described in any one of claims 1-8.

Images