Model post-training quantification method and system based on Pareto optimum

Abstract

The invention belongs to the technical field of deep learning, and particularly discloses a model post-training quantification method and system based on Pareto optimization. Through the application, the weight and activation in the pre-trained floating point model are converted into integers with low bit width, the size and precision of the model are simultaneously optimized by utilizing the Pareto thought, and the comprehensive loss of multiple targets is minimum by constructing the joint loss function and calculating the Pareto plane, so that a Pareto optimal state is achieved; and then a balance point with the shortest actual reasoning time is screened out as a final quantification result, so that the improvement of the compression ratio of the model and the reasoning acceleration on resource-limited equipment are realized on the premise of ensuring the precision of the model.

Description

Technical Field

[0001] The present application belongs to the field of deep learning technology, and more specifically, to a model post-training quantization method and system based on Pareto optimality. Background Art

[0002] Model post-training quantization (PTQ) technology is an important model reasoning deployment acceleration technology. It reduces the model's memory usage and computational complexity by reducing the number of weights and activation bits in the model, improves the network reasoning speed, and enables the model to be deployed on resource-constrained edge devices. The advantages of post-training quantization are short quantization time, low resource overhead, low data dependence, simple operation, good compatibility, and can be applied to a variety of different models and frameworks. However, since the quantization error is not adapted during the training phase, the quantization accuracy of the model is poor compared to the quantization method of quantization-aware training (QAT).

[0003] Patent CN2022101013573 discloses a neural network post-training quantization method based on fine-grained data distribution alignment. The specific steps are: calculate the BN parameters of each layer of the pre-trained full-precision network for each image in the calibration data set, and obtain the representative BN parameters of each category; quantize the pre-trained full-precision network to obtain a quantized network; the generator accepts Gaussian noise and randomly selected labels to generate pictures of specified categories, and uses cross entropy loss, BN loss, representative BN loss, and perturbation representative BN loss to train the generator; use the pictures generated by the generator and the pictures in the calibration data set to train the quantized network using distillation loss and cross entropy loss; loop the above steps until the predetermined number of training rounds is reached; retain the weights of the quantized network to obtain the quantized network.

[0004] However, existing model post-training quantization methods cannot balance model size and inference speed while ensuring model accuracy. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to provide a model post-training quantization method and system based on Pareto optimality, aiming to solve the problem that the existing model post-training quantization methods are unable to balance the model size and inference speed while ensuring the model accuracy.

[0006] To achieve the above objectives, in a first aspect, the present application provides a model post-training quantization method based on Pareto optimality, comprising:

[0007] S1. Obtain the ratio of each quantitative target and the calibration data set, and generate a multi-objective joint loss function according to the ratio of each quantitative target, wherein the quantitative target includes model compression ratio and accuracy;

[0008] S2. Obtain the network weight data of the pre-trained floating-point model, use the pre-trained floating-point model to perform forward reasoning on the images in the calibration data set, and obtain the network activation data of each activation layer during the forward reasoning process;

[0009] S3. quantize the weight data and the activation data to obtain several candidate quantization models, wherein at least one of the quantization strategies and quantization bit widths of each layer of different candidate quantization models is different;

[0010] S4. Input the images in the calibration data set into each candidate quantization model for forward reasoning, calculate the multi-objective joint loss function, and the quantization strategies and quantization bit widths corresponding to all candidate quantization models in the minimum state constitute the Pareto optimal boundary;

[0011] S5. Sequentially select equilibrium points from the Pareto optimal frontier, perform model quantification based on the selected equilibrium points, and obtain several preliminary screening quantitative models;

[0012] S6. Use each preliminary screening quantitative model to perform actual task reasoning, and count the reasoning accuracy of each preliminary screening quantitative model. If there is a preliminary screening quantitative model whose reasoning accuracy is not lower than the preset requirement, the preliminary screening quantitative model with the shortest actual reasoning time will be used as the final quantitative model. Otherwise, adjust the proportion of each quantitative target, regenerate the multi-target joint loss function, and enter step S3.

[0013] Preferably, the multi-objective joint loss function is as follows:

[0014] Loss = α·Loss θ +β·Loss(x)

[0015] Among them, Loss represents the total loss, α and β are the ratio of model compression ratio and accuracy respectively, and Loss θ It represents the loss function of the model compression ratio, Loss(x) represents the loss function of the quantized model inference, and x represents the input of the model.

[0016] Preferably, the loss function of the model compression ratio is as follows:

[0017]

[0018] Among them, K represents the total number of layers in the network, θ i Indicates the quantization bit width of this layer.

[0019] Preferably, the quantization strategy is symmetric quantization, asymmetric quantization, linear quantization or nonlinear quantization.

[0020] Preferably, the quantization bit width is 1 bit, 2 bits, 4 bits, 8 bits or 16 bits.

[0021] To achieve the above-mentioned purpose, in a second aspect, the present application provides a model post-training quantization system based on Pareto optimality, comprising at least one processor and at least one memory;

[0022] The at least one memory is used to store computer instructions;

[0023] The at least one processor is configured to execute at least part of the computer instructions to implement the method described in the first aspect.

[0024] To achieve the above objectives, in a third aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the method described in the first aspect.

[0025] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0026] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0027] This application proposes a model post-training quantization method and system based on Pareto optimality, which converts the weights and activations in the pre-trained floating-point model into integers with low bit width, and uses the Pareto idea to simultaneously optimize the model size and accuracy. By constructing a joint loss function and calculating the Pareto plane, the comprehensive loss of multiple objectives is minimized and a Pareto optimal state is achieved. Then, the balance point with the shortest actual reasoning time is selected as the final quantization result, thereby achieving an improvement in the model compression ratio and accelerating reasoning on resource-constrained devices while ensuring the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a model post-training quantization method based on Pareto optimality provided in an embodiment of the present application.

[0029] Figure 2 It is a schematic diagram of the Pareto frontier determination process provided in an embodiment of the present application.

[0030] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0033] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0034] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0035] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0036] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0037] like Figure 1 As shown, the present application provides a model post-training quantization method based on Pareto optimality, including:

[0038] S1. Obtain the ratio of each quantitative target and a calibration data set, and generate a multi-objective joint loss function according to the ratio of each quantitative target, wherein the quantitative targets include model compression ratio and accuracy.

[0039] The model post-training quantization method proposed in this application can be applied to different usage scenarios. If it is a vehicle recognition scenario, the calibration data set is a license plate image set; if it is a lane detection scenario, the calibration data set is a lane line image set; if it is a text recognition scenario, the calibration data set is a text image set. The ratio of each quantization target is set according to the usage scenario. The larger the ratio value, the more important the quantization target.

[0040] Preferably, the multi-objective joint loss function is as follows:

[0041] Loss = α·Loss θ +β·Loss(x)

[0042] Among them, Loss represents the total loss, α and β are the ratio of model compression ratio and accuracy respectively, and Loss θ It represents the loss function of the model compression ratio, Loss(x) represents the loss function of the quantized model inference, and x represents the input of the model.

[0043] Add parameters θ to different network layers of the original model i , which is used to represent the quantization bit width of the layer and construct the loss function of the model compression ratio.

[0044] Preferably, the loss function of the model compression ratio is as follows:

[0045]

[0046] Among them, K represents the total number of layers in the network, θ i Indicates the quantization bit width of this layer.

[0047] If the model is used for classification tasks, the loss function for quantizing model reasoning is the cross entropy loss; if the model is used for detection tasks, the loss function for quantizing model reasoning is a combination of MAE and cross entropy loss or a combination of MSE and cross entropy loss.

[0048] S2. Obtain the network weight data of the pre-trained floating-point model, use the pre-trained floating-point model to perform forward inference on the images in the calibration dataset, and obtain the network activation data of each activation layer during the forward inference process.

[0049] Quantizing weights generally does not require calibration data; quantizing activation values ​​requires calibration data.

[0050] S3. Quantize the weight data and activation data to obtain several candidate quantization models, where at least one of the quantization strategies and quantization bit widths of each layer is different between different candidate quantization models.

[0051] Preferably, the quantization strategy is symmetric quantization, asymmetric quantization, linear quantization or nonlinear quantization.

[0052] Preferably, the quantization bit width is 1 bit, 2 bits, 4 bits, 8 bits or 16 bits.

[0053] S4. Input the images in the calibration dataset into each candidate quantization model for forward reasoning, and calculate the multi-objective joint loss function. The quantization strategies and quantization bit widths corresponding to all candidate quantization models in the minimum state constitute the Pareto optimal boundary.

[0054] Pareto optimality represents an ideal state in a multi-objective optimization task. Under Pareto optimality, it is impossible to optimize any of the multiple objectives without compromising other objectives. In the process of model quantization, it is necessary not only to consider the model size and inference speed of the quantized model, but also to ensure the inference accuracy of the model. However, the size and accuracy of the model often conflict with each other. In order to balance multiple objectives in the process of model quantization, this application incorporates the idea of ​​Pareto optimality into the process of model quantization. By constructing a Pareto optimization model, the multi-objective losses are combined into an overall loss function for multi-objective optimization. Figure 2 As shown in the figure, by seeking the Pareto boundary of the model compression ratio index and the model accuracy index, the quantized model achieves the optimal balance between resource consumption, inference speed and accuracy.

[0055] Optimize the joint loss function and calculate the quantization bit width θ when Loss is at the minimum value i According to θ i Quantize the network layer corresponding to the original model to achieve mixed precision quantization of the model.

[0056] S5. Sequentially select equilibrium points from the Pareto optimal frontier, perform model quantification based on the selected equilibrium points, and obtain several preliminary screening quantitative models.

[0057] S6. Use each preliminary screening quantitative model to perform actual task reasoning, and count the reasoning accuracy of each preliminary screening quantitative model. If there is a preliminary screening quantitative model whose reasoning accuracy is not lower than the preset requirement, the preliminary screening quantitative model with the shortest actual reasoning time will be used as the final quantitative model. Otherwise, adjust the proportion of each quantitative target, regenerate the multi-target joint loss function, and enter step S3.

[0058] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0059] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0060] like Figure 3 As shown, based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory 830 communicate with each other through the communication bus. The processor can call the logic instructions in the memory to execute the method in the above embodiment.

[0061] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0062] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0063] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0064] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0065] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0066] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0067] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0068] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model post-training quantization method based on Pareto optimality, characterized in that: include: S1. Obtain the ratio of each quantitative target and the calibration data set, and generate a multi-objective joint loss function according to the ratio of each quantitative target, wherein the quantitative target includes model compression ratio and accuracy; S2. Obtain the network weight data of the pre-trained floating-point model, use the pre-trained floating-point model to perform forward reasoning on the images in the calibration data set, and obtain the network activation data of each activation layer during the forward reasoning process; S3. quantize the weight data and the activation data to obtain several candidate quantization models, wherein at least one of the quantization strategies and quantization bit widths of each layer of different candidate quantization models is different; S4. Input the images in the calibration data set into each candidate quantization model for forward reasoning, calculate the multi-objective joint loss function, and the quantization strategies and quantization bit widths corresponding to all candidate quantization models in the minimum state constitute the Pareto optimal boundary; S5. Sequentially select equilibrium points from the Pareto optimal frontier, perform model quantification based on the selected equilibrium points, and obtain several preliminary screening quantitative models; S6. Use each preliminary screening quantitative model to perform actual task reasoning, and count the reasoning accuracy of each preliminary screening quantitative model. If there is a preliminary screening quantitative model whose reasoning accuracy is not lower than the preset requirement, the preliminary screening quantitative model with the shortest actual reasoning time will be used as the final quantitative model. Otherwise, adjust the proportion of each quantitative target, regenerate the multi-target joint loss function, and enter step S3.

2. The method according to claim 1, characterized in that The multi-objective joint loss function is as follows: Loss=α·Loss θ +β·Loss(x) Among them, Loss represents the total loss, α and β are the ratio of model compression ratio and accuracy respectively, and Loss θ It represents the loss function of the model compression ratio, Loss(x) represents the loss function of the quantized model inference, and x represents the input of the model.

3. The method according to claim 2, characterized in that The loss function of the model compression ratio is as follows: Among them, K represents the total number of layers in the network, θ i Indicates the quantization bit width of this layer.

4. The method according to claim 1, characterized in that The quantization strategy is symmetric quantization, asymmetric quantization, linear quantization or nonlinear quantization.

5. The method according to claim 1, characterized in that The quantization bit width is 1 bit, 2 bits, 4 bits, 8 bits or 16 bits.

6. A model post-training quantization system based on Pareto optimality, characterized in that: comprising at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least part of the computer instructions to implement the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the method according to any one of claims 1 to 5.

Images