Quantitative tuning device and method suitable for AI chip
By splitting the model into subgraphs and combining tuning strategies, and performing quantitative parameter optimization one by one, the problems of model accuracy loss and low deployment efficiency in the existing technology are solved, and better model compression and acceleration effects are achieved.
Patent Information
- Application Number
- CN202410954897.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
Existing neural network quantization technologies, especially PTQ and QAT, have problems with model accuracy loss and low deployment efficiency. PTQ requires manual adjustment of hyperparameters, while QAT requires a large amount of calibration data and the quantization operation is slow.
By splitting the model into multiple subgraphs, combining the tuning strategy using a combination method, tuning each subgraph one by one, and trying different quantization parameters, which are updated only when the quantization loss is better than the original parameters, so as to achieve better results model compression and acceleration effects.
While maintaining model accuracy, better model compression and acceleration effects are achieved, and the required data amount is similar to PTQ, avoiding the high calibration data requirements of QAT.
Smart Images

Figure CN118917365A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a quantization tuning device and method suitable for AI chips. Background Art
[0002] With the continuous development of AI technology, dedicated AI chips for accelerating artificial intelligence computing have been widely used. AI chips are embedded with neural network models. At the same time, in order to meet the computing power requirements, model compression is required. Among them, neural network quantization is one of the most effective ways to achieve these requirements.
[0003] At present, most neural network quantization acceleration algorithms use PTQ (Post-Training Quantization) and QAT (Quantization-Aware Training) quantization technologies.
[0004] PTQ quantization technology is a method of quantizing a model after training. Its advantages include requiring less calibration data and faster quantization speed, which allows for faster model deployment. PTQ achieves model compression and acceleration by converting the trained model into a low-precision representation, such as integer or binary.
[0005] QAT quantization technology makes the model robust to quantization errors by introducing quantization operations during the training process. QAT can be implemented by simulating quantization operations or using hardware-supported quantization functions. QAT can effectively reduce the accuracy loss caused by quantization.
[0006] However, PTQ may result in significant loss of model accuracy, and in some cases may require manual tuning of hyperparameters to optimize performance; while QAT usually requires a large amount of calibration data, and the quantization operation is relatively slow, affecting deployment efficiency. Summary of the invention
[0007] In view of the technical defects mentioned in the background technology, the purpose of the embodiments of the present invention is to provide a quantization tuning device and method suitable for AI chips, so as to achieve better model compression and acceleration effects while taking into account the model accuracy.
[0008] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a quantization tuning device applicable to an AI chip, comprising:
[0009] A splitting module, used to split the model into multiple subgraphs according to constraints; wherein the model is a deep learning floating-point model, and the constraints are limitations faced by the AI chip during the quantization process;
[0010] The setting module is used to combine a number of tuning settings in a combined manner to obtain a corresponding tuning strategy;
[0011] Processing modules for:
[0012] Performing tuning processing on each subgraph in turn in combination with the tuning strategy and judging the effect;
[0013] If the result is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
[0014] As a preferred implementation of the present application, the tuning settings include search settings, subgraph observation layer settings, subgraph input settings, and definitions of evaluation indicators used for the observation layer of each subgraph and how to compare corresponding indicators.
[0015] As a preferred implementation of the present application, the search setting includes:
[0016] Only tune the activation Scale / Zero Point of some layers with poor calibration effect;
[0017] Tune the activation scale / zero point of all layers;
[0018] Tune the Scale / Zero Point of activations and weights of all layers.
[0019] As a preferred implementation of the present application, the sub-image observation layer setting is used to specify an observation layer for the sub-image, and the observation layer includes:
[0020] Subgraph output;
[0021] The next level of subgraph;
[0022] The lower N layers of the subgraph; where N is a positive integer;
[0023] Model output.
[0024] As a preferred implementation of the present application, the quantization tuning device suitable for an AI chip further includes a tuner module, and the tuner module is used to combine a plurality of tuners in a combined manner.
[0025] As a preferred implementation of the present application, different quantization parameters will be tried during the tuning process; and the original quantization parameter will be replaced only when the quantization parameter makes the quantization loss of the entire model or sub-graph better than before the tuning.
[0026] In a second aspect, an embodiment of the present invention further provides a quantization tuning method applicable to an AI chip, which is applied to a quantization tuning device applicable to an AI chip described in the first aspect, and the method includes:
[0027] Split the model into multiple subgraphs according to constraints; wherein the model is a deep learning floating-point model, and the constraints are limitations faced by the AI chip during the quantization process;
[0028] A combination approach is adopted to combine many tuning settings to obtain the corresponding tuning strategy;
[0029] Performing tuning processing on each subgraph in turn in combination with the tuning strategy and judging the effect;
[0030] If the result is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
[0031] The technical solution provided by the embodiment of the present invention adopts a modular design structure, which is convenient for different models to adopt different combinations to obtain corresponding tuning strategies; at the same time, the model is divided into multiple sub-graphs according to constraints and tuned one by one, different quantization parameters are tried and updated selectively to achieve better tuning effects; compared with the existing PTQ and QAT quantization technologies, the amount of data required by this technical solution is similar to that of PTQ, and it can achieve better model compression and acceleration effects while maintaining model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.
[0033] Figure 1 This is a principle block diagram of a quantization tuning device applicable to an AI chip provided by an embodiment of the present invention;
[0034] Figure 2 This is a schematic diagram of an application provided by an embodiment of the present invention;
[0035] Figure 3 It is a flow chart of a quantitative tuning method suitable for an AI chip provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0038] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any appropriate combinations and / or subcombinations.
[0039] It should be noted that, unless otherwise specified, the technical terms in this embodiment have the common meanings understood in the relevant technical field.
[0040] PTQ: Post-Training Quantization, or post-training quantization, is a technique for converting a model from floating-point representation to fixed-point representation to reduce the storage and computational overhead of the model.
[0041] QAT: Quality Aware Training. This is a technique that takes quantization errors into account during training, aiming to improve the accuracy of the model after quantization.
[0042] Tuning: refers to the process of adjustment or optimization. Tuning means fine-tuning the quantization parameters after the regular PTQ calibration to find better parameters.
[0043] Scale: The scaling factor used in quantization to map floating-point values to integer values.
[0044] Zero Point: The zero point value in quantization, used to offset integer values.
[0045] Quantization parameters: Scale and Zero Point.
[0046] Quantization loss: refers to the difference in accuracy between the quantized model and the original floating-point model.
[0047] Symmetric Quantization: Symmetric Quantization is a quantization technique in which the Zero Point is 0.
[0048] Asymmetric quantization: Asymmetric quantization is a quantization technique in which the zero point may not be 0. The range of values that can be represented may be different in the positive and negative ranges.
[0049] Q-value quantization: Q-value quantization is a quantization technique in which the quantization parameter Scale needs to be an integer power of 2. Usually, Q-value quantization is limited by hardware constraints for the purpose of faster inference. However, the accuracy after quantization is usually lower than the accuracy after Scale quantization without Q-value quantization.
[0050] AI chip: refers to a dedicated chip used to accelerate artificial intelligence computing.
[0051] Similarity: refers to the cosine similarity, the cosine value of the angle between two vectors, which is used to evaluate the difference between the model before and after quantization.
[0052] MSE: Mean-Square Error (MSE) is a measure of the difference between the estimator and the estimated quantity.
[0053] Subgraph: A part of a model.
[0054] Please refer to Figure 1 , an embodiment of the present invention provides a quantization tuning device suitable for an AI chip, comprising:
[0055] A splitting module, used to split the model into multiple subgraphs according to constraints; wherein the model is a deep learning floating-point model, and the constraints are limitations faced by the AI chip during the quantization process;
[0056] The setting module is used to combine a number of tuning settings in a combined manner to obtain a corresponding tuning strategy;
[0057] Processing modules for:
[0058] Performing tuning processing on each subgraph in turn in combination with the tuning strategy and judging the effect;
[0059] If the result is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
[0060] It should be noted that, refer to Figure 2 Tuning is an optional operation as a post-calibration optimization scheme in PTQ. If the accuracy of the model after quantitative calibration has met the requirements, the Tuning operation can be skipped. If the accuracy does not meet the requirements, the Tuning operation can be added after calibration.
[0061] When applied, the constraints refer to the restrictions or requirements faced by AI chips during the quantization process. For example, in order to accelerate reasoning, some layers use Q value quantization or some layers require the Scale and Zero Point to be consistent. Each subgraph has the constraint that the activation response Scale and Zero Point must be consistent, and the weights in the subgraph may have different Scales and ZeroPoints.
[0062] Other hardware constraints may also be included, such as the memory capacity, computing power, and memory bandwidth of the AI chip, which are only examples and are not intended to limit them.
[0063] For the constraints of different AI chips, better tuning effects can be achieved after fine-tuning the corresponding setting files.
[0064] In this embodiment, the tuning settings include search settings, subgraph observation layer settings, subgraph input settings, and define the evaluation indicators used for the observation layer of each subgraph and how to compare the corresponding indicators; the settings are performed through the Strategies class.
[0065] The search settings include:
[0066] Only tune the activation Scale / Zero Point of some layers with poor calibration effect;
[0067] Tune the activation scale / zero point of all layers;
[0068] Tune the Scale / Zero Point of activations and weights of all layers;
[0069] Users can also customize their search habits according to their needs and choose the desired search settings.
[0070] The sub-image observation layer setting is used to specify an observation layer for a sub-image, and the observation layer includes:
[0071] Subgraph output;
[0072] The next level of subgraph;
[0073] The lower N layers of the subgraph; where N is a positive integer;
[0074] Model output;
[0075] This defines how to specify the observation layer for a sub-graph, and defines the corresponding observation layer for different sub-graphs; similarly, users can also customize the observation layer definition habits according to their needs.
[0076] The input setting of the subgraph is implemented by using the ActvationProviderBehavior in the Strategies class, which defines whether the input of each subgraph comes from the subgraph input of the fp32 model or the subgraph input of the quantized model; users can also customize the corresponding habits according to their needs;
[0077] Furthermore, it defines how to manage and provide activation data, including pre-extracting fp32 model activations to save time, online inference of fp32 and quanted models, etc.
[0078] Defines the evaluation indicators used for the observation layer of each sub-graph and how to compare the corresponding indicators, including:
[0079] A single indicator or multiple indicators can be applied to each sub-graph; currently, the evaluation indicators include common indicators such as cos and mse, and users can also customize corresponding evaluation habits according to their needs.
[0080] The Subgraphs class module also records the corresponding Scale / Zero Point of the activation and weight, and records the observation layer of the specified subgraph.
[0081] Furthermore, during implementation, it also includes a tuner module, and the tuner module is used to combine a plurality of tuners in a combined manner.
[0082] Through the provided Optimizers class, the weight and activation tuners are derived. Using a combination method, many tuners are combined, including (asymmetric), Q-value, BF16, and Index-type tuners; among them, (asymmetric) and Q-value tuners both need to adjust both weights and activations; BF16 and Index-type tuning do not perform actual tuning operations, only adjust weights, and do not need to adjust activations; the activation can be understood as the dynamic data generated according to the actual application of the model.
[0083] Through the above settings, it is easy for users to choose to combine or expand the advantages of different tuning strategies; thereby overcoming the poor scalability and ease of use of the existing tuning framework, which is not conducive to users adjusting or expanding different tuning strategies according to the actual situation of the model.
[0084] The split module is also used to update the Scale / Zero Point specified by the Optimizers according to the constraints, so as to adapt to different platform architectures.
[0085] In this embodiment, different quantization parameters are tried during the tuning process; and the original quantization parameter is replaced only when the quantization parameter makes the quantization loss of the entire model or sub-graph better than before the tuning (that is, the quantization loss is smaller);
[0086] When applied, it is further judged whether the preset maximum number of attempts is reached, and when the preset maximum number of attempts is reached and it is the last sub-graph, the quantization tuning is terminated.
[0087] At the same time, in Tuning, the tuning under the constraints of the AI chip is taken into consideration; while conventional PTQ calibration, after appropriate adjustment, only meets the constraints, but does not necessarily obtain better quantization parameters under the constraints.
[0088] The above scheme adopts a modular design structure, which makes it convenient to use different combinations for different models to derive corresponding tuning strategies; at the same time, the model is divided into multiple sub-graphs according to the constraints and tuned one by one, and different quantization parameters are tried and updated selectively to achieve better tuning effects; compared with the existing PTQ and QAT quantization technologies, the amount of data required by this technical scheme is similar to that of PTQ, and it can achieve better model compression and acceleration effects while maintaining model accuracy.
[0089] Reference Figure 3 Based on the same inventive concept, an embodiment of the present invention further provides a quantization tuning method applicable to an AI chip, which is applied to a quantization tuning device applicable to an AI chip described in the first aspect, and the method comprises the following steps:
[0090] S101, splitting the model into multiple subgraphs according to the constraint conditions; wherein the model is a deep learning floating-point model, and the constraint conditions are limitations faced by the AI chip during the quantization process;
[0091] S102, adopting a combination method, combining a plurality of tuning settings to obtain a corresponding tuning strategy;
[0092] S103, performing tuning processing on each subgraph in turn in combination with the tuning strategy and determining the effect;
[0093] S104: If the result of the judgment is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
[0094] Furthermore, the tuning settings include search settings, subgraph observation layer settings, subgraph input settings, and definitions of evaluation indicators used for the observation layer of each subgraph and how to compare the corresponding indicators.
[0095] Furthermore, the method also includes: combining a plurality of tuners using a combination method.
[0096] During the tuning process, different quantization parameters will be tried, and the original quantization parameters will be replaced only when the quantization parameters make the quantization loss of the entire model or subgraph better than before the tuning.
[0097] When applied, it is further judged whether the preset maximum number of attempts is reached, and when the preset maximum number of attempts is reached and it is the last sub-graph, the quantization tuning is terminated.
[0098] It should be noted that for a more specific description of the workflow of the method embodiment, please refer to the aforementioned device embodiment part, which will not be repeated here.
[0099] The entire solution makes it easy to use different combinations for different models to derive corresponding tuning strategies. At the same time, the model is divided into multiple sub-graphs according to constraints and tuned one by one. Different quantization parameters are tried and updated selectively to achieve better tuning effects. Compared with the existing PTQ and QAT quantization technologies, the amount of data required by this technical solution is similar to that of PTQ, and it can achieve better model compression and acceleration effects while maintaining model accuracy.
[0100] In the embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic. It should also be noted that in some alternative implementations, the functions marked in the blocks can also occur in an order different from the order marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0101] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0102] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0103] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A quantitative tuning device suitable for AI chips, characterized in that: include: A splitting module, used to split the model into multiple subgraphs according to constraints; wherein the model is a deep learning floating-point model, and the constraints are limitations faced by the AI chip during the quantization process; The setting module is used to combine a number of tuning settings in a combined manner to obtain a corresponding tuning strategy; Processing modules for: Performing tuning processing on each subgraph in turn in combination with the tuning strategy and judging the effect; If the result is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
2. A quantization tuning device suitable for an AI chip according to claim 1, characterized in that: The tuning settings include search settings, subgraph observation layer settings, subgraph input settings, and define the evaluation indicators used for the observation layer of each subgraph and how to compare the corresponding indicators.
3. A quantization tuning device suitable for an AI chip as claimed in claim 2, characterized in that: The search settings include: Only tune the activation Scale / Zero Point of some layers with poor calibration effect; Tune the activation scale / zero point of all layers; Tune the Scale / Zero Point of activations and weights of all layers.
4. The quantization tuning device suitable for AI chip according to claim 2, characterized in that: The sub-image observation layer setting is used to specify an observation layer for a sub-image, and the observation layer includes: Subgraph output; The next level of subgraph; The lower N layers of the subgraph; where N is a positive integer; Model output.
5. A quantization tuning device suitable for an AI chip according to any one of claims 1 to 4, characterized in that: It also includes a tuner module, which is used to combine multiple tuners in a combined manner.
6. A quantization tuning device suitable for an AI chip as claimed in claim 5, characterized in that: During the tuning process, different quantization parameters will be tried, and the original quantization parameter will be replaced only when the quantization parameter makes the quantization loss of the entire model or subgraph better than before the tuning.
7. A quantitative tuning method suitable for AI chips, characterized in that: The method for the quantization tuning device applicable to an AI chip as claimed in claim 1 comprises: Split the model into multiple subgraphs according to constraints; wherein the model is a deep learning floating-point model, and the constraints are limitations faced by the AI chip during the quantization process; A combination approach is adopted to combine many tuning settings to obtain the corresponding tuning strategy; Performing tuning processing on each subgraph in turn in combination with the tuning strategy and judging the effect; If the result is better than that before the optimization and it is the last sub-image, the quantization optimization is terminated and the corresponding quantization parameters are updated.
8. The quantitative tuning method applicable to an AI chip according to claim 6, characterized in that: The tuning settings include search settings, subgraph observation layer settings, subgraph input settings, and define the evaluation indicators used for the observation layer of each subgraph and how to compare the corresponding indicators.
9. A quantitative tuning method suitable for an AI chip as claimed in claim 7 or 8, characterized in that: The method also includes: combining a plurality of tuners using a combination method.
10. A quantitative tuning method suitable for AI chips according to claim 9, characterized in that: During the tuning process, different quantization parameters will be tried, and the original quantization parameter will be replaced only when the quantization parameter makes the quantization loss of the entire model or subgraph better than before the tuning.
Citation Information
Patent Citations
Quantitative model optimization method and device, storage medium and electronic device
CN113469345A
Model processing method and related device
CN114626284A
Deep learning model reasoning method and device, equipment and storage medium
CN115600676A
Deep learning reasoning acceleration method based on graph segmentation
CN115796265A
Neural network quantization method, device and equipment
CN115952855A
