Quantization method, device, storage medium, and electronic device

Abstract

The present application provides a quantization method, which includes the following steps: obtaining a target operator and quantization data corresponding to the target operator; obtaining quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator; performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator. The quantization method provided by the present application reduces the amount of computation and reduces excessive repeated operations, is convenient to use, improves the quantization efficiency, and optimizes the user experience.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a quantization method, apparatus, storage medium, and electronic device. Background Art

[0002] In recent years, deep neural networks have shown excellent performance in image classification, detection, etc. However, with the improvement of network performance, the computational complexity of the model during training and inference has been continuously increasing, resulting in complex deployment of the model on central nodes or edge devices and limited inference speed.

[0003] Model quantization is a method for compressing a model. This method can reduce the model size and the storage space occupied by the model, reduce memory consumption, and accelerate the inference speed, and is one of the effective techniques for optimizing the model. In the commonly used TensorFlow framework in deep learning, two quantization tools, TensorFlow-Lite and TensorRT, have been provided. However, when using the above two quantization tools for model quantization, not only do we need to write functions separately for different models for data preprocessing, but also the above two quantization tools only support models in two formats, saved_model or frozen graph_def. Users need to convert the model format before quantization, which requires a large amount of repetitive work when users use the quantization tools for model quantization, resulting in poor user experience and low quantization efficiency. Summary of the Invention

[0004] This application provides a quantization method and related devices, which effectively solve the problem of a large amount of repetition required for model quantization in the TensorFlow framework.

[0005] In a first aspect, an embodiment of this application provides a quantization method, and the method includes: obtaining a target operator and quantization data corresponding to the target operator; obtaining quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator; performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0006] In a second aspect, an embodiment of this application provides a quantization apparatus, and the apparatus includes: an obtaining unit, configured to obtain a target operator and quantization data corresponding to the target operator; the obtaining unit is further configured to obtain quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator; a processing unit, configured to perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0007] In a third aspect, an embodiment of the present application provides a quantization device, including: a processor and a memory, and the processor executes the code in the memory to execute the method provided in any implementation manner of the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which when running on a computer, cause the computer to execute the method provided in any implementation manner of the first aspect.

[0009] In the embodiment of the present application, quantization is divided into two stages: obtaining quantization parameters and quantization inference. In the stage of obtaining quantization parameters, the computation graph is directly modified to obtain quantization parameters corresponding to multiple operators to be quantized, and the quantization parameters are saved in the quantization parameter manager. In the stage of quantization inference, the computation graph is directly modified, and quantization inference is performed according to the quantization parameters. In this way, the quantization parameters only need to be calculated once and saved in the quantization parameter manager. Each time the model to be quantized is inferred, only the corresponding quantization parameters need to be obtained from the quantization parameter manager, and there is no need to recalculate the quantization parameters, reducing the computational amount. At the same time, in the embodiment of the present application, by directly modifying the computation graph corresponding to the model, there is no need to consider data preprocessing and there is no need to convert the model type into two formats supported by TensorFlow-Lite or TensorRT, namely saved_model or frozen graph_def, reducing excessive repetitive operations, being convenient to use, improving the quantization efficiency, and optimizing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings required to be used in the embodiments of the present application or the background art will be described below.

[0011] Figure 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0012] Figure 2a is a schematic flowchart of a quantization method provided by an embodiment of the present application;

[0013] Figure 2b is a schematic flowchart of another quantization method provided by an embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a computation graph corresponding to a model to be quantized provided by an embodiment of the present application;

[0015] Figure 4 is a schematic diagram of the modification of a computation graph corresponding to a model to be quantized provided by an embodiment of the present application;

[0016] Figure 5It is a schematic diagram of the modification of the computation graph corresponding to another model to be quantified provided by an embodiment of the present application;

[0017] Figure 6 It is a schematic diagram of a quantization device provided by an embodiment of the present application;

[0018] Figure 7 It is a schematic diagram of the modification of the computation graph corresponding to another model to be quantified provided by an embodiment of the present application;

[0019] Figure 8 [[ID=

[12] ]]It is a structural diagram of a combined processing device provided by an embodiment of the present application;

[0020] Figure 9 It is a schematic diagram of the structure of a board card provided by an embodiment of the present application. Detailed implementation manners

[0021] The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present invention, rather than aiming to limit the present invention.

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] The terms "first", "second", and "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", and "third" may explicitly or implicitly include one or more of such features. It should be understood that when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0024] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0025] Note: There seems to be a small error in the original text where "

[12] " is used instead of "12" in the ID. This has been corrected in the translation.It should also be further understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] First, the application scenarios involved in this application are described.

[0027] In recent years, deep neural networks have shown excellent performance in aspects such as image classification and detection. However, with the improvement of network performance, the computational complexity of the model during training and inference has been continuously increasing, resulting in complex deployment of the model on the central node or edge device and limited inference speed. Model quantization is a method for compressing the model. This method can reduce the model size and the storage space occupied by the model, reduce memory consumption, and speed up the inference speed. It is one of the effective techniques for optimizing the model.

[0028] During the model quantization process of the deep neural network model, two quantization tools, TensorFlow-Lite and TensorRT, have been provided in the commonly used TensorFlow framework in deep learning. When users use the above two quantization tools to quantize the model, first, they need to consider data preprocessing. For example, when the data to be quantized is an image, operations such as cropping and scaling need to be performed on the image. Moreover, for different models, users need to write functions separately to convert the format of the model to be quantized into two formats, saved_model or frozen graph_def, supported by TensorFlow-Lite and TensorRT, and then use the quantization algorithm to perform quantization inference on the model. This process requires a large number of repetitive processes, resulting in poor user experience and low quantization efficiency. The embodiment of this application provides a quantization method. This method directly quantizes the model by modifying the computational graph corresponding to the model to be quantized during the quantization parameter acquisition stage and the quantization inference stage. For different models, users only need to modify very little code or even do not need to modify the code to complete model quantization, and it can support models in formats such as ckpt and h5. It also does not need to consider data preprocessing, providing a convenient quantization method, reducing excessive repetitive operations, improving quantization efficiency, and optimizing the user experience.

[0029] The embodiments of this application are introduced in detail below.

[0030] Please refer to Figure 1 , Figure 1 which is a schematic structural diagram of an electronic device provided by the embodiment of this application. As Figure 1As shown in the figure, the electronic device 100 includes a processor 110, an input device 120, an output device 130, and a memory 140. Among them, the electronic device 100 may further include a communication bus 150, and the processor 110, the input device 120, the output device 130, and the memory 140 may be interconnected through the bus.

[0031] Among them, when the above-mentioned processor 110 executes the program stored in the memory 140, the following steps are implemented:

[0032] Obtain a target operator and quantization data corresponding to the target operator from the memory, and obtain quantization parameters corresponding to the target operator from the quantization parameter manager according to the target operator; perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0033] Furthermore, the above-mentioned processor 110 may be a Central Processing Unit (CPU), an Intelligence Processing Unit (NPU), a Graphics Processing Unit (GPU), or an Image Processing Unit, and this application does not make any limitations in this regard. According to the different processors, the method for determining hardware performance proposed in the embodiments of this application can be applied to artificial intelligence application fields such as image recognition processing, deep learning processing, computer vision processing, intelligent robot processing, and natural language processing, and execute complex functional programs in the field of artificial intelligence.

[0034] The embodiments of this application provide a quantization method, and this method is applied to Figure 1 the electronic device 100 and its processor 110 therein. This method includes: obtaining a target operator and quantization data corresponding to the target operator; obtaining quantization parameters corresponding to the target operator from the quantization parameter manager according to the target operator; performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0035] Please refer to Figure 2a , Figure 2a which is a schematic flowchart of a quantization method provided by this application, and this method is applied to an electronic device processor. As Figure 2a shown in the figure, this method includes the following steps:

[0036] S101. Obtain a target operator and quantization data corresponding to the target operator.

[0037] Specifically, when the model to be quantized performs inference, obtain the target operator and the data to be quantized corresponding to the target operator, where the target operator is one of multiple operators to be quantized in the model to be quantized, and the data to be quantized includes one or more of the input data and weights of the target operator, and the input data includes one or more of speech data, text data, and image data.

[0038] S102. Obtain the quantization parameters corresponding to the target operator from the quantization parameter manager according to the target operator.

[0039] Among them, the quantization parameter manager is used to save quantization parameters. In the quantization parameter manager, the quantization parameters are saved in the form of key-value pairs. The key in the key-value pair is the quantization parameter identifier of the operator to be quantized, and the value in the key-value pair is the content of the quantization parameter. According to the target operator, obtain the quantization parameters corresponding to the target operator from the quantization parameter manager correspondingly.

[0040] S103. Perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the data to be quantized corresponding to the target operator.

[0041] Specifically, in the quantization inference stage, by adding a graph optimizer, modify the computation graph. The modified computation graph retains all the functions of the original computation graph, and when the inputs are the same, the output values of the two are the same. In addition to having the functions of the original computation graph, the modified computation graph also adds the functions of calculating and saving quantization parameters. The modified computation graph can perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the data to be quantized corresponding to the target operator.

[0042] In a possible implementation manner, before performing the calculation, add a second graph optimizer, enable the second graph optimizer, register a third operator, and replace the target operator with the third operator, where the third operator includes the target operator. During inference, first, the third operator can quantize the data to be quantized corresponding to the target operator according to the quantization parameters corresponding to the target operator to obtain the quantized data; then the third operator performs quantization inference according to the quantized data.

[0043] Please refer to Figure 2b , Figure 2b which is a schematic flowchart of another quantization method provided by this application. This method is applied to the processor of an electronic device. As Figure 2b shown, this method includes the following steps:

[0044] S201. Obtain a quantization instruction, a calibration data set, and multiple operators to be quantized.

[0045] Among them, the quantization instruction includes the save path for the quantization parameter manager to store quantization parameters. Specifically, the user can specify the quantization parameter save path by configuring the environment variable. The multiple operators to be quantized are all quantizable operators in the model to be quantized, and the model to be quantized can be a model with training convergence.

[0046] In a possible implementation, the operator to be quantized can be one or more of a local response normalization (LRN) operator, a Conv2D operator, a MatMul operator, and a Conv2DbackpropInput operator, or can be other operators, and the embodiments of the present application do not make any limitations thereto.

[0047] In a possible implementation, the calibration dataset includes the input data corresponding to the operator to be quantized and the weight corresponding to the operator to be quantized. Among them, the input data can be selected from the validation dataset of the model to be quantized and can be one or more of voice data, text data, and image data, and the embodiments of the present application do not make any limitations thereto.

[0048] S202: Obtain the quantization parameter corresponding to each operator to be quantized according to each operator to be quantized among the multiple operators to be quantized and the calibration dataset, and save the quantization parameter in the quantization parameter manager.

[0049] Specifically, in TensorFlow, a model network is constructed using a computational graph, that is, the graph is the model. The computational graph includes multiple operators to be quantized in the model. In the embodiments of the present application, before calculation, a graph optimizer custom_optimizer is added to modify the computational graph. The modified computational graph retains all the functions of the original computational graph, and when the inputs are the same, the output values are the same. In addition to having all the functions of the original computational graph, the modified computational graph also adds the functions of calculating and saving quantization parameters. When the modified computational graph performs calculations, it can calculate the quantization parameter corresponding to the input of each operator to be quantized according to the calibration dataset and each operator to be quantized, and obtain the quantization parameter corresponding to each operator to be quantized, that is, the quantization parameter of the input data corresponding to the operator to be quantized and the quantization parameter of the weight corresponding to the operator to be quantized. After obtaining the quantization parameter corresponding to each operator to be quantized, the quantization parameter corresponding to each operator to be quantized is saved in the quantization parameter manager in the form of key-value pairs.

[0050] In a possible implementation, the quantization parameters may include a scaling factor "scale" and a decimal point position "position", where the decimal point position is the position of the decimal point in the quantized data. The scaling factor is the ratio between the maximum value of the quantized data and the maximum absolute value of the data to be quantized. Further, the quantization parameters may also include an offset, which is for asymmetric data to be quantized and refers to the median value of multiple elements in the data to be quantized. Specifically, the offset may be the midpoint value of multiple elements in the data to be quantized.

[0051] In a possible implementation, the modified computation graph calculates the quantization parameters corresponding to the input of each operator to be quantized according to the calibration dataset and each operator to be quantized. Specifically, it can be:

[0052]

[0053] where max_abs is the maximum absolute value of the calibration dataset corresponding to the operator to be quantized, position is the decimal point position, scale is the scaling factor, and ceil represents rounding up. The calculation method exemplified in the embodiments of this application is only for illustration, and the calculation method of the quantization parameters by the modified computation graph may also be other methods, and the embodiments of this application do not make any limitations in this regard.

[0054] In a possible implementation, before performing the calculation, a first graph optimizer is added and the first graph optimizer is enabled. For each operator to be quantized among multiple operators to be quantized, a corresponding first operator is registered, and each operator to be quantized is replaced with the corresponding first operator. Among them, the enabling and disabling of the graph optimizer can be set through an environment variable flag, or can be through setting tf.ConfigProto of tensorflow, where tf.ConfigProto is a configurable parameter of tensorflow, or can be other ways to enable or disable, and the embodiments of this application do not make any limitations in this regard.

[0055] Exemplarily, as Figure 3 shown in the neural network model, including a ReLU activation operator, a Conv2D convolution operator, and a BN batch normalization operator. If the operator to be quantized is the Conv2D operator, taking the Conv2D operator as an example for illustration, the first operator CambQuantConv2D corresponding to the Conv2D operator is registered, and the CambQuantConv2D operator is used to replace the Conv2D operator, as Figure 4As shown in the figure. The CambQuantConv2D operator can calculate the quantization parameters of the Conv2D operator according to the calibration dataset, and save the quantization parameters in the quantization parameter manager in the form of key-value pairs. Among them, the key in the key-value pair is the quantization parameter identifier of the operator to be quantized, and the value in the key-value pair is the content of the quantization parameter. For example, in the quantization parameter manager, the quantization parameter corresponding to the Conv2D operator is saved in the form of scale_map{key conv0 / input-v 1.7, key conv0 / filter-v 3.6, key conv1 / input-v 6.7, key conv1 / filter-v 7.7}.

[0056] In a possible implementation, the above first operator can be a single operator or a combination of multiple operators. When the first operator is a combination of multiple operators, the first operator includes any combination and all possible combinations of the corresponding operator to be quantized, the operator for calculating quantization parameters, and the operator for saving quantization parameters. The embodiments of the present application do not make any limitation on this. Among them, the operator for calculating quantization parameters is used to calculate the quantization parameters of the operator to be quantized corresponding to the first operator, and the operator for saving quantization parameters is used to save the quantization parameters of the operator to be quantized corresponding to the first operator in the save path of the quantization parameter manager according to the quantization instruction.

[0057] In a possible implementation, before performing the calculation, add the first graph optimizer and enable the first graph optimizer, register the corresponding second operator for each operator to be quantized among the multiple operators to be quantized, and insert the second operator corresponding to each operator to be quantized in front of the corresponding operator to be quantized.

[0058] Exemplarily, as Figure 3 shown in the neural network model, taking the Conv2D operator as an example for illustration, register the second operator CambQuant corresponding to the Conv2D operator, and insert the CambQuant operator in front of the Conv2D operator. As Figure 5 shown, the neural network model calculates the Conv2D operator after calculating the CambQuant operator. The CambQuant operator can calculate the quantization parameters of the Conv2D operator according to the calibration dataset and save the quantization parameters in the quantization parameter manager.

[0059] In a possible implementation, the above-mentioned second operator may be a single operator for calculating and storing quantization parameters; the second operator may also be two operators, including an operator for calculating quantization parameters and an operator for storing quantization parameters. The embodiments of the present application do not make any limitations in this regard. Among them, the operator for calculating quantization parameters is used to calculate the quantization parameters of the operator to be quantized corresponding to the first operator, and the operator for storing quantization parameters is used to store the quantization parameters of the operator to be quantized corresponding to the first operator under the storage path of the quantization parameter manager according to the quantization instruction.

[0060] In the embodiments of the present application, by dividing model quantization into a quantization parameter acquisition stage and a quantization inference stage, in the quantization parameter acquisition stage, by directly modifying the computation graph before performing the calculation, the quantization parameters of the operator to be quantized can be obtained during the calculation, and the quantization parameters are stored in the quantization parameter manager. In this way, the quantization parameters only need to be calculated once in the quantization parameter acquisition stage, and there is no need to calculate the quantization parameters every time when the model input data is different, reducing the computation amount.

[0061] S203. Obtain the target operator and the data to be quantized corresponding to the target operator, and obtain the quantization parameters corresponding to the target operator from the quantization parameter manager according to the target operator.

[0062] Specifically, during the inference of the model to be quantized, the target operator and the data to be quantized corresponding to the target operator are obtained. The target operator is one of the multiple operators to be quantized in the model to be quantized, and the data to be quantized includes one or more of the input data and weights of the target operator. The input data includes one or more of voice data, text data, and image data. According to the target operator, the quantization parameters corresponding to the target operator are retrieved from the quantization parameter manager.

[0063] Exemplarily, in the quantization parameter manager, the storage form of the quantization parameters corresponding to the Conv2D operator is scale_map{key conv0 / input–v 1.7, key conv0 / filter–v 3.6, key conv1 / input–v 6.7, keyconv1 / filter–v 7.7}. According to the operator conv0, the quantization parameter of the corresponding input data can be obtained as 1.7, and the quantization parameter of the weight is 3.6. According to the operator conv1, the quantization parameter of the corresponding input data can be obtained as 6.7, and the weight parameter is 7.7.

[0064] S204. Perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the data to be quantized corresponding to the target operator.

[0065] Specifically, in the quantization inference stage, by adding a graph optimizer, the computational graph is modified. The modified computational graph retains all the functions of the original computational graph, and when the inputs are the same, the output values of the two are the same. In addition to having the functions of the original computational graph, the modified computational graph also adds the functions of calculating and saving quantization parameters. The modified computational graph can perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the data to be quantized corresponding to the target operator.

[0066] In a possible implementation, before performing the calculation, a second graph optimizer is added and the second graph optimizer is enabled, a third operator is registered, and the target operator is replaced with the third operator. During inference, first, the third operator can quantize the data to be quantized corresponding to the target operator according to the quantization parameters corresponding to the target operator to obtain the quantized data; then the third operator performs quantization inference according to the quantized data.

[0067] In a possible implementation, the above-mentioned third operator can be a single operator or a combination of multiple operators. When the third operator is a combination of multiple operators, the third operator includes any combination and all possible combinations of the corresponding target operator and quantization operators, and the embodiments of the present application do not make any limitations thereto. Among them, the quantization operator is used to quantize the data to be quantized.

[0068] In a possible implementation, the specific calculation method for the third operator to quantize the data to be quantized corresponding to the target operator according to the quantization parameters corresponding to the target operator is:

[0069]

[0070] F x ≈I x ×2 position ×scale(4)

[0071] Where, I x represents the data after fixed-point quantization, and F x represents the data before quantization. The specific method for the third operator in the embodiments of the present application to quantize the data to be quantized corresponding to the target operator according to the quantization parameters corresponding to the target operator is only an example, and the specific calculation method can also be other calculation methods, and the present application does not make any limitations thereto.

[0072] In the embodiments of the present application, quantization refers to converting data in a first format into data in a second format. The data in the first format can be floating-point data or fixed-point data, and the data in the second format can be fixed-point data. The representation range of the data in the first format is greater than that of the data in the second format, and the representation of the data in the first format has higher precision than that of the data in the second format. For example, the data in the first format can be 64-bit floating-point data or 32-bit floating-point data, and the data in the second format can be 16-bit fixed-point data or 8-bit fixed-point data. Quantization can be converting 64-bit floating-point data into 16-bit fixed-point data or 8-bit fixed-point data, or converting 32-bit floating-point data into 16-bit fixed-point data or 8-bit fixed-point data. It can also be conversions of other data types, and the embodiments of the present application do not make any limitations in this regard.

[0073] In a possible implementation manner, as Figure 3 shown by the neural network model, the following takes the target operator as the Conv2D operator as an example for illustration. When the model performs inference, the second graph optimizer is enabled, the third operator, the intConv2D operator, is registered, and the Conv2D operator is replaced with the intConv2D operator, as Figure 6 shown. The intConv2D operator can quantize the quantized data according to the quantization parameters corresponding to the Conv2D operator to obtain the quantized data, and perform inference based on the quantized data.

[0074] In a possible implementation manner, the first graph optimizer and the second graph optimizer can be the same optimizer or different optimizers.

[0075] In summary, in the embodiments of the present application, quantization is divided into two stages: obtaining quantization parameters and quantization inference. In the stage of obtaining quantization parameters, the computation graph is directly modified to obtain the quantization parameters corresponding to multiple operators to be quantized, and the quantization parameters are saved in the quantization parameter manager. In the stage of quantization inference, the computation graph is directly modified to perform quantization inference according to the quantization parameters. In this way, the quantization parameters only need to be calculated once and saved in the quantization parameter manager. Each time the model to be quantized performs inference, only the corresponding quantization parameters need to be obtained from the quantization parameter manager, without having to recalculate the quantization parameters, reducing the computation amount. At the same time, in the embodiments of the present application, by directly modifying the computation graph corresponding to the model, there is no need to consider data preprocessing and there is no need to convert the type of the model into the two formats of saved_model or frozen graph_def supported in TensorFlow-Lite or TensorRT, reducing excessive repetitive operations, being convenient to use, improving the quantization efficiency, and optimizing the user experience.

[0076] The above mainly introduced the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that in order for an electronic device to implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0077] Embodiments of the present application can divide the functional units of an electronic device according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0078] Please refer to Figure 7 , Figure 7 is a functional unit composition block diagram of a device 700 for determining hardware performance provided by an embodiment of the present application. As Figure 7 shown, the device 700 for determining hardware performance includes an acquisition unit 710 and a processing unit 720. Among them,

[0079] The acquisition unit 710 is configured to acquire a target operator and the quantization data corresponding to the target operator, and acquire the quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator;

[0080] The processing unit 720 is configured to perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0081] In a specific embodiment of the quantization device of the present invention, the specific operations of the above acquisition unit 710 to acquire a target operator and the quantization data corresponding to the target operator and acquire the quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator can refer to the relevant operations in S203 above, which will not be elaborated here; the relevant operations of the above processing unit 720 to perform quantization inference on the target operator can refer to the relevant operations in S204 above, which will not be elaborated here.

[0082] In a specific implementation manner, the above-mentioned acquisition unit 710 is further configured to acquire quantization instructions, a calibration data set, and a plurality of operators to be quantized. The specific operations of the acquisition unit 710 for acquiring quantization instructions, a calibration data set, and a plurality of operators to be quantized may refer to the relevant operations in S201 above, which will not be elaborated here. The above-mentioned processing unit 720 is further configured to obtain quantization parameters corresponding to each of the plurality of operators to be quantized according to each operator to be quantized in the plurality of operators to be quantized and the calibration data set, and save the quantization parameters corresponding to each quantization operator under the saving path of the quantization parameter manager according to the quantization instructions. The specific operations of the processing unit 720 for obtaining quantization parameters and saving the quantization parameters under the saving path of the quantization parameter manager may refer to the relevant operations in S202 above, which will not be elaborated here.

[0083] It can be understood that the functions of the respective program modules of the device for determining hardware performance in the embodiments of the present application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions in the above method embodiments, which will not be elaborated here.

[0084] The embodiments of the present application further provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute some or all of the steps of any of the methods described in the above method embodiments.

[0085] The embodiments of the present application further provide a computer program product. The above computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above computer program is operable to cause a computer to execute some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package.

[0086] Figure 8 FIG. is a structural diagram showing a combined processing device 800 according to an embodiment of the present disclosure. As Figure 8 shown, the combined processing device 800 includes a computing processing device 802, an interface device 804, other processing devices 806, and a storage device 808. According to different application scenarios, the computing processing device may include one or more computing devices 810, and the computing device may be configured to execute the operations described herein in connection with the appendix Figure 2a and Figure 2b described.

[0087] In various embodiments, the computing processing device of the present disclosure can be configured to perform operations specified by a user. In an exemplary application, the computing processing device can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included within the computing processing device can be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, the computing processing device of the present disclosure can be regarded as having a single-core structure or a homogeneous multi-core structure.

[0088] In an exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through an interface device to jointly complete the operations specified by the user. Depending on the implementation, the other processing devices of the present disclosure can include one or more types of general-purpose and / or special-purpose processors such as a central processing unit (CPU), a graphics processing unit (GPU), and an artificial intelligence processor. These processors can include, but are not limited to, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and the number thereof can be determined according to actual needs. As mentioned above, only with respect to the computing processing device of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when considering the computing processing device and other processing devices together, the two can be regarded as forming a heterogeneous multi-core structure.

[0089] In one or more embodiments, the other processing device can serve as an interface for external data and control of the computing processing device of the present disclosure (which can be specifically implemented as an operation device related to artificial intelligence such as neural network operations), and perform basic controls including but not limited to data transfer, turning on and / or off of the computing device. In additional embodiments, the other processing device can also cooperate with the computing processing device to jointly complete an operation task.

[0090] In one or more embodiments, the interface device can be used to transfer data and control instructions between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it into the storage device (or memory) on the chip of the computing processing device. Further, the computing processing device can obtain control instructions from other processing devices via the interface device and write them into the control cache on the chip of the computing processing device. Alternatively or optionally, the interface device can also read the data in the storage device of the computing processing device and transfer it to other processing devices.

[0091] Additionally or optionally, the combined processing device of the present disclosure may further include a storage device. As Figure 8 shown, the storage device is respectively connected to the computing processing device and the other processing device. In one or more embodiments, the storage device can be used to store the data of the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage device of the computing processing device or other processing devices.

[0092] In some embodiments, the present disclosure also discloses a neural network chip (such as Figure 9 the chip 902 shown). In one implementation, the chip is a System on Chip (SoC) and integrates one or more combined processing devices as Figure 7 shown. The chip can be connected to other related components through an external interface device (such as the external interface device 806 shown in Figure 8 ). The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. In some application scenarios, other processing units (such as a video codec) and / or interface modules (such as a DRAM interface) etc. can be integrated on the chip. In some embodiments, the present disclosure also discloses a chip package structure that includes the above chip. In some embodiments, the present disclosure also discloses a board that includes the above chip package structure. The following will be described in detail with reference to Figure 8 the board.

[0093] Figure 9 is a schematic structural diagram of a board 900 according to an embodiment of the present disclosure. As Figure 9As shown, the board includes a storage device 904 for storing data, which includes one or more storage units 910. The storage device can be connected to and transfer data with a control device 908 and the chip 902 described above through, for example, a bus. Further, the board also includes an external interface device 906, which is configured to perform data relay or transfer functions between the chip (or chips in the chip package structure) and an external device 912 (such as a server or a computer). For example, the data to be processed can be transmitted from the external device to the chip through the external interface device. Another example is that the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms. For example, it can adopt a standard PCIE interface, etc.

[0094] In one or more embodiments, the control device in the disclosed board can be configured to regulate the state of the chip. For this purpose, in one application scenario, the control device can include a microcontroller unit (MCU) for regulating the operating state of the chip.

[0095] According to the above combination Figure 8 and Figure 9 of the description, those skilled in the art can understand that the present disclosure also discloses an electronic device or apparatus, which can include one or more of the above boards, one or more of the above chips, and / or one or more of the above combined processing devices.

[0096] According to different application scenarios, the electronic devices or apparatuses of the present disclosure may include servers, cloud servers, server clusters, data processing apparatuses, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, earphones, mobile storages, wearable devices, vision terminals, autonomous driving terminals, transportation means, household appliances, and / or medical devices. The transportation means includes airplanes, ships, and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical devices include nuclear magnetic resonance imagers, B-ultrasound devices, and / or electrocardiographs. The electronic devices or apparatuses of the present disclosure can also be applied to fields such as the Internet, Internet of Things, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical. Further, the electronic devices or apparatuses of the present disclosure can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing such as the cloud, edge, and terminal. In one or more embodiments, the electronic devices or apparatuses with high computing power according to the solution of the present disclosure can be applied to cloud devices (such as cloud servers), while the electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of cloud devices is compatible with the hardware information of terminal devices and / or edge devices, so that appropriate hardware resources can be matched from the hardware resources of cloud devices according to the hardware information of terminal devices and / or edge devices to simulate the hardware resources of terminal devices and / or edge devices, so as to complete unified management, scheduling, and collaborative work of end-cloud integration or cloud-edge-end integration.

[0097] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and their embodiments as a series of actions and their combinations. However, those skilled in the art can understand that the solution of the present disclosure is not limited by the order of the described actions. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art can understand that some of the steps can be executed in other orders or simultaneously. Further, those skilled in the art can understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved are not necessarily required for the implementation of a certain or certain solutions of the present disclosure. In addition, according to the differences in the solutions, the present disclosure also focuses on the descriptions of some embodiments. In view of this, those skilled in the art can understand that the parts not detailed in a certain embodiment of the present disclosure can also refer to the relevant descriptions of other embodiments.

[0098] In terms of specific implementation, based on the disclosure and teachings herein, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, regarding each unit in the foregoing embodiments of the electronic device or apparatus, they are divided herein based on consideration of logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. Regarding the connection relationship between different units or components, the connections discussed in conjunction with the accompanying drawings herein can be direct or indirect couplings between the units or components. In some scenarios, the foregoing direct or indirect couplings involve communication connections using interfaces, where the communication interfaces can support signal transmissions in electrical, optical, acoustic, magnetic, or other forms.

[0099] In this disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The foregoing components or units may be located at the same position or distributed across multiple network units. Additionally, according to actual needs, some or all of the units can be selected to achieve the objectives of the solutions described in the embodiments of this disclosure. Further, in some scenarios, multiple units in the embodiments of this disclosure can be integrated into one unit or each unit physically exists separately.

[0100] In some implementation scenarios, the above-mentioned integrated units can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as an independent product, the integrated units can be stored in a computer-readable memory. Based on this, when the solution of this disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in the memory, which may include several instructions for causing a computer device (such as a personal computer, server, or network device, etc.) to execute some or all of the steps of the method described in the embodiments of this disclosure. The foregoing memory may include, but is not limited to, various media such as USB flash drives, flash memory drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0101] In some other implementation scenarios, the above integrated units can also be implemented in the form of hardware, that is, a specific hardware circuit, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to devices such as transistors or memristors. In view of this, various devices described in this article (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, a Resistive Random Access Memory (RRAM), a Dynamic Random Access Memory (DRAM), a Static Random Access Memory (SRAM), an Enhanced Dynamic Random Access Memory (EDRAM), a High Bandwidth Memory (HBM), a Hybrid Memory Cube (HMC), a ROM, and a RAM, etc.

[0102] The foregoing can be better understood in accordance with the following clauses:

[0103] Clause A1. A quantization method, comprising:

[0104] Obtaining a target operator and quantization data corresponding to the target operator;

[0105] According to the target operator, obtaining quantization parameters corresponding to the target operator from a quantization parameter manager;

[0106] Performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator.

[0107] Clause A2. The method according to Clause A1, before the obtaining the target operator and quantization data corresponding to the target operator, the method further comprises:

[0108] Obtaining a quantization instruction, wherein the quantization instruction includes a save path for the quantization parameter manager to store quantization parameters;

[0109] Obtaining a calibration data set and a plurality of operators to be quantized, and the target operator is one of the plurality of operators to be quantized;

[0110] For each quantization operator among the multiple quantization operators to be quantized and the calibration data set, obtain the quantization parameter corresponding to each quantization operator, and save the quantization parameter corresponding to each quantization operator in the saving path of the quantization parameter manager according to the quantization instruction.

[0111] Clause A3. The method according to Clause A2, for each quantization operator among the multiple quantization operators to be quantized and the calibration data set, obtain the quantization parameter corresponding to each quantization operator, and save the quantization parameter corresponding to each quantization operator in the saving path of the quantization parameter manager according to the quantization instruction, includes:

[0112] Register a corresponding first operator for each quantization operator to be quantized, where the first operator includes the quantization operator to be quantized corresponding to the first operator;

[0113] Obtain the quantization parameter corresponding to each quantization operator according to the first operator corresponding to each quantization operator and the calibration data set;

[0114] According to the quantization instruction, the first operator corresponding to each quantization operator saves the quantization parameter corresponding to each quantization operator in the quantization parameter manager.

[0115] Clause A4. The method according to Clause A2, for each quantization operator among the multiple quantization operators to be quantized and the calibration data set, obtain the quantization parameter corresponding to each quantization operator, and save the quantization parameter corresponding to each quantization operator in the saving path of the quantization parameter manager according to the quantization instruction, includes:

[0116] Register a corresponding second operator for each quantization operator to be quantized;

[0117] Determine the quantization parameter corresponding to each quantization operator according to the second operator corresponding to each quantization operator and the calibration data set;

[0118] According to the quantization instruction, the second operator corresponding to each quantization operator saves the quantization parameter corresponding to each quantization operator in the quantization parameter manager.

[0119] Clause A5. The method according to Clause A1, the quantized inference of the target operator according to the quantization parameter corresponding to the target operator and the data to be quantized corresponding to the target operator, includes:

[0120] Register a third operator, where the third operator includes a target operator, and the third operator is used to quantize the data to be quantized according to the quantization parameters corresponding to the target operator to obtain quantized data, so that the third operator performs inference based on the quantized data.

[0121] The third operator performs quantization inference according to the quantization parameters to be quantized corresponding to the target operator.

[0122] Clause A6. According to the method described in Clause A5, the method is applied to the TensorFlow architecture, the data to be quantized includes one or more of the input data and weights of the target operator, and the data to be input includes one or more of voice data, text data, and image data.

[0123] A7. A quantization device, comprising:

[0124] An acquisition unit, configured to acquire a target operator and the data to be quantized corresponding to the target operator;

[0125] The acquisition unit is further configured to acquire the quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator;

[0126] A processing unit, configured to perform quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the data to be quantized corresponding to the target operator.

[0127] A8. According to the device described in Clause A7, comprising:

[0128] The acquisition unit is further configured to acquire a quantization instruction, where the quantization instruction includes a save path for the quantization parameter manager to store quantization parameters; acquire a calibration data set and a plurality of operators to be quantized, and the target operator is any one of the plurality of operators to be quantized;

[0129] The processing unit is further configured to obtain the quantization parameters corresponding to each operator to be quantized according to each operator to be quantized among the plurality of operators to be quantized and the calibration data set, and save the quantization parameters corresponding to each quantization operator in the save path of the quantization parameter manager according to the quantization instruction.

[0130] A9. A quantization device, comprising: a processor and a memory, and the processor executes the code in the memory to execute the method described in any one of Clauses A1 to A6.

[0131] A10. A computer-readable storage medium, comprising a computer program for data exchange, and when the computer program is executed by a processor, it implements the method described in any one of Clauses A1 - A6.

[0132] Although several embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that the above embodiments are provided by way of example only. Those skilled in the art can conceive of many changes, alterations, and alternative ways without departing from the spirit and scope of the present disclosure. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in practicing the present disclosure. The appended claims are intended to define the scope of protection of the present disclosure and thus cover equivalents or alternatives within the scope of these claims.

Claims

1. A quantization method, characterized in that, The method is applied to the TensorFlow architecture and includes: Obtaining a target operator and quantization data corresponding to the target operator; the quantization data includes one or more of the input data and weights of the target operator, and the input data includes one or more of speech data, text data, and image data; Obtaining quantization parameters corresponding to the target operator from a quantization parameter manager according to the target operator; Performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator; The performing quantization inference on the target operator according to the quantization parameters corresponding to the target operator and the quantization data corresponding to the target operator includes: Adding a graph optimizer, enabling the graph optimizer, and registering a third operator, where the third operator includes the target operator, and the third operator is used to quantize the quantization data according to the quantization parameters corresponding to the target operator to obtain quantized data, so that the third operator performs inference according to the quantized data; the graph optimizer is used to modify the computation graph; the computation graph includes multiple operators to be quantized in the model; The third operator performs quantization inference according to the quantization parameters corresponding to the target operator.

2. The method according to claim 1, wherein Before the obtaining the target operator and the quantization data corresponding to the target operator, the method further includes: Obtaining a quantization instruction, where the quantization instruction includes a save path for the quantization parameter manager to store quantization parameters; Obtaining a calibration data set and multiple operators to be quantized, and the target operator is one of the multiple operators to be quantized; Obtaining quantization parameters corresponding to each operator to be quantized according to each operator to be quantized among the multiple operators to be quantized and the calibration data set; Saving the quantization parameters corresponding to each quantization operator in the save path of the quantization parameter manager according to the quantization instruction.

3. The method according to claim 2, wherein Obtaining quantization parameters corresponding to each operator to be quantized according to each operator to be quantized among the multiple operators to be quantized and the calibration data set includes: Registering a corresponding first operator for each operator to be quantized, where the first operator is used to implement the operation of the corresponding operator to be quantized and to calculate and save the quantization parameters of the corresponding operator to be quantized; Obtaining quantization parameters corresponding to each operator to be quantized according to the first operator corresponding to each operator to be quantized and the calibration data set.

4. The method according to claim 3, wherein The first operator is one operator, and the method further includes: Replacing each operator to be quantized with the first operator corresponding to each operator to be quantized; Obtaining quantization parameters corresponding to each operator to be quantized according to the first operator corresponding to each operator to be quantized and the calibration data set.

5. The method according to claim 3, wherein The first operator further includes at least one of a quantization parameter calculation operator and a quantization parameter saving operator. The quantization parameter calculation operator is configured to calculate quantization parameters of an operator to be quantized corresponding to the first operator, and the quantization parameter saving operator is configured to save the quantization parameters of the operator to be quantized corresponding to the first operator in the saving path of the quantization parameter manager according to the quantization instruction.

6. The method according to claim 2, characterized in that, Obtaining the quantization parameter corresponding to each operator to be quantized according to each operator to be quantized in the plurality of operators to be quantized and the calibration data set includes: Registering a corresponding second operator for each operator to be quantized, where the second operator is configured to save the quantization parameters of the operator to be quantized corresponding to the second operator in the saving path of the quantization parameter manager according to the quantization instruction; Determining the quantization parameter corresponding to each operator to be quantized according to the second operator corresponding to each operator to be quantized and the calibration data set.

7. A quantization device, characterized in that, The apparatus is applied to a TensorFlow architecture and includes: An obtaining unit, configured to obtain a target operator and data to be quantized corresponding to the target operator; the data to be quantized includes one or more of input data and weights of the target operator, and the input data includes one or more of voice data, text data, and image data; The obtaining unit is further configured to obtain the quantization parameter corresponding to the target operator from a quantization parameter manager according to the target operator; A processing unit, configured to perform quantization inference on the target operator according to the quantization parameter corresponding to the target operator and the data to be quantized corresponding to the target operator; Specifically, the processing unit is configured to: add a graph optimizer, enable the graph optimizer, and register a third operator, where the third operator includes the target operator, and the third operator is configured to quantize the data to be quantized according to the quantization parameter corresponding to the target operator to obtain quantized data, so that the third operator performs inference according to the quantized data; the graph optimizer is configured to modify a computation graph; the computation graph includes a plurality of operators to be quantized in a model; the third operator performs quantization inference according to the quantization parameter corresponding to the target operator.

8. The quantization device according to claim 7, characterized in that, Before the obtaining unit is configured to obtain a target operator and data to be quantized corresponding to the target operator, the obtaining unit is further configured to: Obtain a quantization instruction, where the quantization instruction includes a saving path for the quantization parameter manager to store quantization parameters; Obtain a calibration data set and a plurality of operators to be quantized, and the target operator is any one of the plurality of operators to be quantized; The processing unit is further configured to obtain the quantization parameter corresponding to each operator to be quantized according to each operator to be quantized in the plurality of operators to be quantized and the calibration data set, and save the quantization parameter corresponding to each quantization operator in the saving path of the quantization parameter manager according to the quantization instruction.

9. A quantization device, characterized in that, It includes: A processor and a memory, and the processor executes the code in the memory to perform the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, Comprising instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images