Automatic mixed bit quantization device and method suitable for AI chip

By splitting the deep learning model into subgraphs and filtering the sensitive operators to configure them as high bits, the problem of inference taking time in the process of hybrid bit quantization in the existing technology is solved, and efficient model inference and accuracy improvement is achieved, which is suitable for all Mixed bit tuning of models and scenes.

CN118917366APending Publication Date: 2024-11-08BEIJING YIXIN YIYU MICROELECTRONICS TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410954938.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the process of hybrid bit quantization of deep learning models, the inference takes a long time and calculates the Hessian matrix for a large weight matrix, which affects efficiency.

Method used

By converting the deep learning floating-point model into an onnx model and performing graph optimization, splitting it into different subgraphs, filtering the sensitive operators and configuring them as high bits, generating a mixed bit model, and automatically filtering the sensitive computing using the subgraph inference of the quantized model after the quantization model sub, realize the optimal combination of high bits and low bits.

Benefits of technology

It significantly improves inference speed and execution efficiency, saves search time, takes into account model accuracy and performance, and is suitable for mixed bit tuning of various models and scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917366A_ABST
    Figure CN118917366A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic mixed bit quantization device and method suitable for an AI chip. The method comprises the steps that a deep learning floating point model is converted into an onx model, and graph optimization is carried out; performing INT8 quantization on the onnx after graph optimization, and outputting a quantized onnx model; according to the quantization coefficient, splitting the onx model into different sub-graphs, and dividing nodes with the same quantization coefficient into the same sub-graph; for each operator, the input of the operator and the output of the upper layer corresponding to the input are converted into high-bit input and high-bit output, then reasoning is carried out on the sub-graph where the current operator is located, and the difference value before modification and the difference value after modification are compared; sequencing the difference values before and after all operators, selecting K operators with optimal precision improvement according to the preset number K of high-bit operators, configuring the K operators as a high-bit type, and generating a mixed bit configuration table; generating a mixed bit model according to the selected configuration; the method has the advantages that model precision and performance can be considered.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to an automatic mixed bit quantization device and method suitable for AI chips. Background Art

[0002] At present, deep learning models have been widely used. When applying them, a hybrid bit quantization solution can be used to balance model performance and accuracy.

[0003] The more conventional method at present is to reduce the operators of each layer from high bits to low bits in turn, and then infer the entire model to obtain the corresponding accuracy of the model, and sort the sensitivity of the operators according to the accuracy information. If the accuracy of model reasoning of low-bit operators is seriously reduced, they are regarded as sensitive operators. In this way, sensitive operators can be set to high bits, and the remaining operators use low bits. Some other commonly used methods, such as using the Hessian matrix of weights (i.e., the second-order derivative matrix) to evaluate the sensitivity of parameters of different layers, use high bits for layers with large derivative change rates, and use low bits for layers with relatively gentle derivatives.

[0004] However, after each operator is set to low bits, the entire model must be re-inferred, which takes a long time and greatly affects efficiency. For matrices with large weights using the Hessian matrix, calculating the Hessian matrix is ​​also very time-consuming, requiring a lot of exploration time and computing resources. Summary of the invention

[0005] In view of the technical defects mentioned in the background technology, the purpose of the embodiments of the present invention is to provide an automatic mixed bit quantization device and method suitable for AI chips.

[0006] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides an automatic hybrid bit quantization device applicable to an AI chip, which is applied to the AI ​​chip and includes:

[0007] The conversion module is used to convert the acquired deep learning floating-point model into an onnx model and perform graph optimization;

[0008] A quantization module, used to perform INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and output the quantized onnx model; wherein the onnx model includes corresponding model parameters;

[0009] Processing modules for:

[0010] According to the quantization coefficient, the onnx model is split into different subgraphs, and nodes with the same quantization coefficient are divided into the same subgraph; wherein the quantization coefficient is calculated according to the model parameters;

[0011] For each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference values ​​before and after the modification;

[0012] Configuration modules for:

[0013] Sort the difference values ​​before and after all operators, and select K operators with the best accuracy improvement according to the preset number K of high-bit operators, configure them as high-bit types, and generate a mixed bit configuration table;

[0014] Generate a mixed bit model based on the selected configuration for application to the AI ​​chip.

[0015] As a preferred implementation method of the present application, the graph optimization includes merging, deleting, replacing operators, and adding attributes corresponding to the AI ​​chip to the graph.

[0016] As a preferred implementation of the present application, when generating the mixed bit model, a conversion operator is inserted for converting data types between high bits and INT8.

[0017] As a preferred implementation of the present application, generating a mixed bit model according to the selected configuration specifically includes:

[0018] The graph-optimized onnx model and the generated hybrid bit configuration table are passed into the tool chain of the AI ​​processor. The hybrid bit configuration table is parsed by the tool chain and re-quantized using the graph-optimized onnx model to generate a hybrid bit model.

[0019] As a preferred implementation of the present application, the difference value includes the calculated mean square error and cosine similarity.

[0020] In a second aspect, an embodiment of the present invention further provides an automatic hybrid bit quantization method applicable to an AI chip, which is applied to an automatic hybrid bit quantization device applicable to an AI chip described in the first aspect, and the method includes:

[0021] Convert the acquired deep learning floating point model into an onnx model and perform graph optimization;

[0022] Perform INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and output the quantized onnx model; wherein the onnx model includes corresponding model parameters;

[0023] According to the quantization coefficient, the onnx model is split into different subgraphs, and nodes with the same quantization coefficient are divided into the same subgraph; wherein the quantization coefficient is calculated according to the model parameters;

[0024] For each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference values ​​before and after the modification;

[0025] Sort the difference values ​​before and after all operators, and select K operators with the best accuracy improvement according to the preset number K of high-bit operators, configure them as high-bit types, and generate a mixed bit configuration table;

[0026] Generate a mixed bit model based on the selected configuration for application to the AI ​​chip.

[0027] The technical solution provided by the embodiment of the present invention utilizes subgraph reasoning of the quantized model to automatically screen out sensitive operators, configure them as high bits, and generate a mixed bit model, thereby achieving the optimal combination of high bits and low bits, so as to achieve the purpose of both ensuring the accuracy of the model and improving the reasoning speed; at the same time, by utilizing the output accuracy of the model subgraph rather than the output accuracy of the entire model network, there is no need to reason on the entire graph when searching for sensitive operators, which greatly saves the search time and reduces the time of mixed bit search, and can significantly improve the corresponding execution efficiency; and the technical solution does not have high hardware requirements and can be widely used in mixed bit tuning of various models and scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0029] Figure 1 This is a principle block diagram of an automatic hybrid bit quantization device applicable to an AI chip provided by an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of an automatic mixed bit quantization method suitable for an AI chip provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any appropriate combinations and / or subcombinations.

[0034] It should be noted that, unless otherwise specified, the technical terms in this embodiment have the common meanings understood in the relevant technical field.

[0035] Model quantization: A model generally refers to a neural network model for deep learning; quantization refers to converting the weights in a neural network model from floating point (FLOAT32) to fixed point (such as INT8) or other data formats (such as BF16) to compress the model weights and accelerate reasoning. Reasoning refers to the forward calculation of a neural network.

[0036] Hybrid bit quantization: Different from INT8 quantization or BF16 quantization, hybrid bit quantization can configure different data types for a single operator; the hybrid bit of the present invention mainly refers to the mixture of INT8 and BF16 operators to solve the problem of insufficient INT8 quantization accuracy and insufficient BF16 quantization performance, taking into account both model performance and accuracy.

[0037] Pytorch: A Python-based deep learning framework for neural network reasoning and training.

[0038] Onnx: is an open neural network exchange format that allows model interoperability between different deep learning frameworks. It stores the model structure, weight parameters, and output information of each node in a binary format. This format allows the model to run on different deep learning frameworks and hardware platforms while maintaining its accuracy and performance.

[0039] Similarity: refers to the cosine similarity, the cosine value of the angle between two vectors, which is used to evaluate the difference between the model before and after quantization.

[0040] MSE: Mean-Square Error (MSE) is a measure of the difference between the estimator and the estimated quantity.

[0041] Subgraph: A part of a model.

[0042] Please refer to Figure 1 , an automatic hybrid bit quantization device suitable for an AI chip provided by an embodiment of the present invention is applied to the AI ​​chip, comprising:

[0043] The conversion module is used to convert the acquired deep learning floating-point model into an onnx model and perform graph optimization;

[0044] A quantization module, used to perform INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and output the quantized onnx model; wherein the onnx model includes corresponding model parameters;

[0045] Processing modules for:

[0046] According to the quantization coefficient, the onnx model is split into different subgraphs, and nodes with the same quantization coefficient are divided into the same subgraph; wherein the quantization coefficient is calculated according to the model parameters;

[0047] For each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference values ​​before and after the modification;

[0048] Configuration modules for:

[0049] Sort the difference values ​​before and after all operators, and select K operators with the best accuracy improvement according to the preset number K of high-bit operators, configure them as high-bit types, and generate a mixed bit configuration table;

[0050] Generate a mixed bit model based on the selected configuration for application to the AI ​​chip.

[0051] In this embodiment, the deep learning floating-point model is converted into an onnx model by using Pytorch; wherein the tool chain of the AI ​​processor can adopt the Pytorch framework; the high-bit type is illustrated using BF16 as an example, but it is not limited thereto.

[0052] When applied, the graph optimization includes merging, deleting, replacing operators, and adding attributes corresponding to the AI ​​chip to the graph; wherein the attributes here include quantization coefficients, zero points, and displacement biases, etc.;

[0053] The quantization coefficient is calculated according to the model parameters, specifically including:

[0054] Quantization formula expression from int8 to float;

[0055] r = S (qZ)

[0056]

[0057] Among them, r is the real value, float type; q is the quantized value, INT8 type;

[0058] S represents scale, which is the quantization coefficient and is float; Z represents the zero point and is of INT8 type;

[0059] max and min are the maximum and minimum values ​​in the weight;

[0060] When quantized to INT8, qmax and qmin are the maximum and minimum values ​​of the INT8 value range, that is, 127 and -128; Round is a rounding operation;

[0061] Quantization from float to int8 is: q = clamp (Round (r / s) + Z, qmax, qmin)

[0062] Clamp is a truncation operation, which truncates the value to the int8 value range.

[0063] Splitting the onnx model into different subgraphs does not require actual model splitting, but only requires storing node names of the same subgraph;

[0064] The difference values ​​include the calculated mean square error (MSE) and cosine similarity (COS).

[0065] Then, configure each operator as BF16, calculate the MSE and COS of the output of the subgraph where the operator is located and the output of the corresponding subgraph of the floating-point model, and compare them with the output of the original INT8 model subgraph, and calculate the difference between the MSE and COS after the operator is configured as BF16;

[0066] Then sort the MSE and COS differences calculated by all operators, and select operators with significantly improved accuracy as high-bit types; the number of configurations is based on the preset number K of high-bit operators;

[0067] During implementation, the number of high-bit operators can be selected based on the specific accuracy requirements of the model, so as to maximize the balance between the performance and accuracy requirements of the model.

[0068] Further, generating a mixed bit model according to the selected configuration specifically includes:

[0069] The graph-optimized onnx model and the generated hybrid bit configuration table are passed to the tool chain of the AI ​​processor, the hybrid bit configuration table is parsed by the tool chain, and the graph-optimized onnx model is used to re-quantize the hybrid bit model, so as to generate a hybrid bit model;

[0070] At the same time, when generating the mixed bit model, a conversion operator is inserted to convert the data type between high bit and INT8; the conversion operator is a custom function. When converting high bit (BF16) to INT8, the conversion operator will first convert the BF16 operator to float32 type, and then quantize the float32 type to INT8 type. Conversely, when converting INT8 to BF16, the INT8 will be first dequantized to float32 type, and then converted to BF16 type.

[0071] The above scheme uses subgraph reasoning of the quantized model to automatically screen out sensitive operators, configure them as high bits, and generate a mixed bit model, thereby achieving the optimal combination of high bits and low bits, so as to achieve the purpose of both ensuring the accuracy of the model and improving the reasoning speed; at the same time, by using the output accuracy of the model subgraph rather than the output accuracy of the entire model network, there is no need to reason on the entire graph when searching for sensitive operators, which greatly saves search time and reduces the time of mixed bit search, and can significantly improve the corresponding execution efficiency; in addition, this technical solution has low hardware requirements and can be widely used in mixed bit tuning of various models and scenarios.

[0072] The technical solution of the present invention is applicable to various scenarios that require model compression and acceleration, especially deep learning applications on resource-constrained edge computing devices. For example, image recognition, speech recognition, natural language processing and other applications on smartphones, smart home devices, drones and other devices can all use the technical solution of the present invention to optimize the execution efficiency and accuracy of the model.

[0073] Reference Figure 2 Based on the same inventive concept, an embodiment of the present invention further provides an automatic hybrid bit quantization method applicable to an AI chip, which is applied to the automatic hybrid bit quantization device applicable to an AI chip described in the first aspect, and the method comprises the following steps:

[0074] S101, converting the acquired deep learning floating point model into an onnx model and performing graph optimization;

[0075] S102, performing INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and outputting the quantized onnx model; wherein the onnx model includes corresponding model parameters;

[0076] S103, splitting the onnx model into different subgraphs according to the quantization coefficients, and dividing nodes with the same quantization coefficients into the same subgraph; wherein the quantization coefficients are calculated according to the model parameters;

[0077] S104, for each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference value before and after the modification;

[0078] S105, sorting the difference values ​​before and after all operators, and selecting K operators with the best accuracy improvement according to the preset number K of high-bit operators, and configuring them as high-bit types, and generating a mixed bit configuration table;

[0079] S106, generating a mixed bit model according to the selected configuration to be applied to the AI ​​chip.

[0080] Furthermore, the graph optimization includes merging, deleting, replacing operators and adding attributes corresponding to the AI ​​chip to the graph; the difference value includes the calculated mean square error and cosine similarity.

[0081] When generating the mixed bit model, a conversion operator is inserted for converting data types between high bits and INT8.

[0082] It should be noted that for a more specific description of the workflow of the method embodiment, please refer to the aforementioned device embodiment part, which will not be repeated here.

[0083] The entire scheme utilizes the output accuracy of the model subgraph, and does not need to infer the entire graph when searching for sensitive operators, which greatly saves search time. And based on the difference between high-bit output accuracy and low-bit output accuracy, some operators with the most obvious accuracy improvement are selected and configured as high-bit types. This makes the hybrid bit strategy have a great advantage in speed while taking into account the performance of the model.

[0084] In the embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic. It should also be noted that in some alternative implementations, the functions marked in the blocks can also occur in an order different from the order marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0086] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0087] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An automatic hybrid bit quantization device suitable for AI chips, It is characterized in that Applied to AI chips, including: The conversion module is used to convert the acquired deep learning floating-point model into an onnx model and perform graph optimization; A quantization module, used to perform INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and output the quantized onnx model; wherein the onnx model includes corresponding model parameters; Processing modules for: According to the quantization coefficient, the onnx model is split into different subgraphs, and nodes with the same quantization coefficient are divided into the same subgraph; wherein the quantization coefficient is calculated according to the model parameters; For each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference values ​​before and after the modification; Configuration modules for: Sort the difference values ​​before and after all operators, and select K operators with the best accuracy improvement according to the preset number K of high-bit operators, configure them as high-bit types, and generate a mixed bit configuration table; Generate a mixed bit model based on the selected configuration for application to the AI ​​chip.

2. The automatic mixed bit quantization device suitable for an AI chip according to claim 1, It is characterized in that The graph optimization includes merging, deleting, replacing operators, and adding attributes corresponding to the AI ​​chip to the graph.

3. The automatic mixed bit quantization device suitable for an AI chip according to claim 1, It is characterized in that When generating the mixed bit model, a conversion operator is inserted to convert data types between high bits and INT8.

4. The automatic mixed bit quantization device suitable for an AI chip according to claim 1, It is characterized in that Generating a mixed bit model according to the selected configuration specifically includes: The graph-optimized onnx model and the generated hybrid bit configuration table are passed into the tool chain of the AI ​​processor. The hybrid bit configuration table is parsed by the tool chain and re-quantized using the graph-optimized onnx model to generate a hybrid bit model.

5. An automatic mixed bit quantization device suitable for an AI chip according to any one of claims 1 to 4, It is characterized in that The difference values ​​include the calculated mean square error and cosine similarity.

6. An automatic hybrid bit quantization method suitable for AI chips, It is characterized in that The automatic hybrid bit quantization device for an AI chip as claimed in claim 1, wherein the method comprises: Convert the acquired deep learning floating point model into an onnx model and perform graph optimization; Perform INT8 quantization on the graph-optimized onnx on the tool chain of the AI ​​processor, and output the quantized onnx model; wherein the onnx model includes corresponding model parameters; According to the quantization coefficient, the onnx model is split into different subgraphs, and nodes with the same quantization coefficient are divided into the same subgraph; wherein the quantization coefficient is calculated according to the model parameters; For each operator, convert its input and the previous layer output corresponding to the input into high-bit input and output, then perform reasoning on the subgraph where the current operator is located, and compare the difference values ​​before and after the modification; Sort the difference values ​​before and after all operators, and select K operators with the best accuracy improvement according to the preset number K of high-bit operators, configure them as high-bit types, and generate a mixed bit configuration table; Generate a mixed bit model based on the selected configuration for application to the AI ​​chip.

7. The automatic hybrid bit quantization method suitable for an AI chip according to claim 6, It is characterized in that The graph optimization includes merging, deleting, replacing operators, and adding attributes corresponding to the AI ​​chip to the graph.

8. An automatic hybrid bit quantization method suitable for an AI chip as claimed in claim 6 or 7, It is characterized in that The difference values ​​include the calculated mean square error and cosine similarity.

9. The automatic hybrid bit quantization method suitable for AI chip according to claim 8, It is characterized in that When generating the mixed bit model, a conversion operator is inserted to convert data types between high bits and INT8.

Citation Information

Cited By

  • Design method of parameterized asymmetric FIFO (First In First Out) and chip circuit

    CN120278107A