Model quantization method and apparatus
By implementing format conversion, quantization processing, and lightweight inference conversion, the system addresses the issues of quantization efficiency and accuracy between different deep learning model formats, optimizes memory usage and parameter updates, and improves cache utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
- Filing Date
- 2023-03-29
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to achieve efficient and high-precision quantization across different deep learning model formats, resulting in low memory usage, inefficient parameter updates, and low cache utilization.
By acquiring the source and target formats of the model to be quantized, performing format conversion, quantizing the model parameters and performing lightweight inference conversion, using training-aware quantization or post-training quantization methods, and combining the model quantization component for accuracy tuning, it can adapt to various deep learning training frameworks.
It improves the versatility and accuracy of model quantization, reduces memory usage and parameter updates, enhances cache utilization, and is suitable for deep learning models of different model formats.
Smart Images
Figure CN116415689B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a model quantization method and apparatus. Background Technology
[0002] With the rapid development of artificial intelligence technology, deep learning models are widely used in scenarios such as image classification, object detection, and speech recognition. The computational complexity of these models is reflected in both storage space and computation time. To reduce memory consumption and parameter updates, and improve cache utilization, it is usually necessary to quantize the model parameters of deep learning models. Deep learning models typically have different model formats; therefore, improving the universality of model quantization and balancing the quantization efficiency and accuracy of deep learning models has become an important research direction. Summary of the Invention
[0003] This application aims to at least partially address one of the technical problems in the related art.
[0004] The first aspect of this application proposes a model quantization method, including:
[0005] Obtain the source deep learning model to be quantized and the first model format of the source deep learning model;
[0006] Obtain the second model format supported by the quantized target deep learning model, and when the first model format and the second model format are inconsistent, perform format conversion on the source deep learning model to obtain the first candidate deep learning model under the second model format;
[0007] The model parameters of the first candidate deep learning model are quantized to obtain the second candidate deep learning model.
[0008] A lightweight inference transformation is performed on the second candidate deep learning model to obtain the quantized target deep learning model.
[0009] This application can improve the versatility of model quantization, and is applicable to deep learning models of different model formats generated by multiple deep learning training frameworks. It can balance the quantization efficiency and quantization accuracy of deep learning models, reduce memory usage and parameter updates, and improve cache utilization.
[0010] A second aspect of this application provides a model quantization apparatus, comprising:
[0011] The first acquisition module is used to acquire the source deep learning model to be quantized and the first model format of the source deep learning model;
[0012] The second acquisition module is used to acquire the second model format supported by the quantized target deep learning model, and when the first model format is inconsistent with the second model format, to perform format conversion on the source deep learning model to acquire the first candidate deep learning model under the second model format.
[0013] The quantization module is used to quantize the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0014] The conversion module is used to perform lightweight inference conversion on the second candidate deep learning model to obtain the quantized target deep learning model.
[0015] A third aspect of this application provides an electronic device, comprising:
[0016] At least one processor; and
[0017] A memory that is communicatively connected to at least one processor; wherein,
[0018] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the model quantization method provided in the first aspect embodiment of this application.
[0019] A fourth aspect of this application provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause a computer to perform a model quantization method according to a first aspect of this application.
[0020] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the model quantization method provided in the first aspect of this application. Attached Figure Description
[0021] Figure 1 This is a flowchart of a model quantization method according to an embodiment of this application;
[0022] Figure 2 This is a flowchart of a model quantization method according to an embodiment of this application;
[0023] Figure 3 This is a flowchart of a model quantization method according to an embodiment of this application;
[0024] Figure 4 This is a flowchart of a model quantization method according to an embodiment of this application;
[0025] Figure 5 This is a schematic diagram of a model quantization method according to an embodiment of this application;
[0026] Figure 6 This is a schematic diagram of a model quantization method according to an embodiment of this application;
[0027] Figure 7 This is a structural block diagram of a model quantization device according to an embodiment of this application;
[0028] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0030] It's important to note that model quantization involves converting a floating-point model to a fixed-point model using some method. For example, quantizing model parameters from Float-32 to lower bit depths, such as Float-16, INT8, or 1-bit. To achieve greater compression ratios and faster execution speeds, or due to limitations in the computing unit type of the deployment platform, models typically require fixed-point quantization of both the weights and activation values of the entire network model to reduce computational overhead. When processing INT8 integer data, graphics processors use faster and cheaper 8-bit tensor kernels to compute convolutions and matrix multiplications, resulting in higher computational throughput. Reducing the precision of activation and parameter data from 32-bit floating-point values to 8-bit integers can lead to a 4x data reduction. Reduced memory footprint means the model requires less storage space, parameter updates are more frequent, and cache utilization is higher.
[0031] The model quantization method and apparatus of this application are described below with reference to the accompanying drawings.
[0032] Figure 1 This is a flowchart of a model quantization method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:
[0033] S101, Obtain the source deep learning model to be quantized and the first model format of the source deep learning model.
[0034] In this embodiment, the source deep learning model can be a high-precision model already trained under a multi-deep learning framework. The source deep learning model can be a model for image recognition, a model for image classification, a model for speech recognition, etc. This disclosure does not limit this.
[0035] In this embodiment, the model format supported by the quantized target deep learning model is a second model format. The model quantization method in this embodiment can be executed by a server or device that supports the second model format. Optionally, the second model format can be a symbolic mathematics system (TensorFlow) model format. In some implementations, the symbolic mathematics system can be deployed on a server. If the server can run the source deep learning model based on the symbolic mathematics system, it can be determined that the first model format and the second model format are consistent. If the server cannot run the source deep learning model based on the symbolic mathematics system, it can be determined that the first model format and the second model format are inconsistent.
[0036] S102, obtain the second model format supported by the quantized target deep learning model, and when the first model format and the second model format are inconsistent, perform format conversion on the source deep learning model to obtain the first candidate deep learning model under the second model format.
[0037] In order to enable the current server or device to support the operation and quantization of the source deep learning model, in this embodiment of the application, when the first model format and the second model format are inconsistent, the source deep learning model is converted to obtain the first candidate deep learning model under the second model format, so as to facilitate the subsequent quantization of the deep learning model.
[0038] In some implementations, the source deep learning model in the first model format can be converted to a second model format based on an intermediate format (ONNX format) to obtain the first candidate deep learning model in the second model format. Optionally, the source deep learning model can be subjected to Open Neural Network Exchange (ONNX) to obtain the third candidate deep learning model in the standard model format. The third candidate deep learning model is then converted from the standard model format to the second model format to obtain the first candidate deep learning model. The standard model format is the ONNX format.
[0039] S103, quantize the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0040] In this model, each parameter can be one or more of the model's weights, biases, and activation values. After format conversion, the model parameters remain unchanged; that is, the model parameters of the source deep learning model are identical to those of the first candidate deep learning model.
[0041] In this embodiment, to balance the quantization efficiency and accuracy of the deep learning model, the model parameters of the first candidate deep learning model can be subjected to training-aware quantization (QAT) or post-training quantization (PTQ) according to the quantization accuracy requirements of the business scenario to obtain the second candidate deep learning model. It should be noted that the model parameters of the first candidate deep learning model can be quantized according to a preset target data bit width, where the target data bit width is the length corresponding to the quantized model parameters. For example, if the quantized model parameters are of type INT8, then the target data bit width is 8 bits.
[0042] It should be noted that the bit width of each model parameter in the source deep learning model is greater than the target bit width. For example, if the model parameters in the model to be compressed are Float-32 (i.e., the bit width is 32 bits), then the quantized model parameters can be of type INT8 (i.e., the target bit width is 8 bits), or they can be of type INT4 (i.e., the target bit width is 4 bits).
[0043] S104, perform lightweight inference transformation on the second candidate deep learning model to obtain the quantized target deep learning model.
[0044] In this embodiment, a lightweight inference transformation (TFLite transformation) is performed on the second candidate deep learning model to obtain the quantized target deep learning model. For the second candidate deep learning model, the data type of the model output can be easily converted using TFLite transformation. Therefore, the target deep learning model can achieve various model quantization functions, such as model data quantization and model output quantization.
[0045] In this embodiment, a source deep learning model to be quantized and a first model format of the source deep learning model are obtained; a second model format supported by the quantized target deep learning model is obtained; and when the first model format and the second model format are inconsistent, the source deep learning model is converted to obtain a first candidate deep learning model under the second model format. This improves the versatility of model quantization and is applicable to deep learning models with different model formats generated by multiple deep learning training frameworks. The model parameters of the first candidate deep learning model are quantized to obtain a second candidate deep learning model; and a lightweight inference conversion is performed on the second candidate deep learning model to obtain the quantized target deep learning model. This balances the quantization efficiency and accuracy of the deep learning model, reduces memory usage and parameter updates, and improves cache utilization.
[0046] Figure 2 This is a flowchart of a model quantization method according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:
[0047] S201, Obtain the source deep learning model to be quantized and the first model format of the source deep learning model.
[0048] S202, obtain the second model format supported by the quantized target deep learning model, and when the first model format and the second model format are inconsistent, perform format conversion on the source deep learning model to obtain the first candidate deep learning model under the second model format.
[0049] For details regarding steps S201 to S202, please refer to the above embodiments; they will not be repeated here.
[0050] S203, obtain the target quantization accuracy of the source deep learning model.
[0051] In some implementations, the target quantization precision of the source deep learning model is the number of decimal places retained during the calculation process. For example, in image data processing, the number of decimal places retained for each channel's pixel values. In other implementations, the business scenario's requirement for quantization precision of the source deep learning model can be obtained. Based on this requirement, the target quantization precision of the source deep learning model can be determined. For instance, the target quantization precision of the source deep learning model can be determined based on the business scenario's requirement for quantization precision, whether high-precision quantization is required or not.
[0052] S204, Invoke the model quantization component, which quantizes the model parameters of the first candidate deep learning model according to the target quantization accuracy to obtain the second candidate deep learning model.
[0053] In this embodiment, the model quantization component can be a precision tuning tool (TFmot quantization tool). For example, if the target quantization precision is greater than or equal to a preset quantization precision threshold, meaning the target quantization precision of the source deep learning model requires high-precision quantization, the model quantization component performs training-aware training (QAT) quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model. For QAT, the TFmot quantization tool can be used for automatic quantization. The advantage of this tool is that it can specify specific layers (e.g., fully connected neural network layers (Dense layers), activation layers) for quantization, effectively quantizing weights and activation values and flexibly combining quantization operations of each layer. It allows for convenient customized quantization of the model, combining flexibility and effectiveness. While QAT reduces model training speed, it can achieve higher precision.
[0054] Optionally, training-aware quantization is performed after the deep learning model has been trained. It involves modifying the structure of the deep learning model by adding some quantization nodes, and then retraining the already trained model. The quantization error noise is incorporated into the model and re-tuned, so almost lossless accuracy can be achieved.
[0055] For example, if the target quantization precision is less than a preset quantization precision threshold, meaning the target quantization precision of the source deep learning model is not high-precision quantization, the model quantization component performs post-training quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model. For Post-training quantization (PTQ), the TFmot quantization tool can be used for automatic quantization. This tool allows specifying the quantization model as a specific data type (e.g., Float16, INT8). PTQ is performed after the high-precision model has been trained. For example, if the model activation values and weights are of type float32, post-training quantization performs INT8 quantization on the weights and activation values of the deep learning model. Weight quantization is relatively easy because the distribution of its values is already known. Activation values are only perceived during inference, so scale calculations using a sample set are necessary. The final model size will be reduced by a factor of 4, but there is a loss of inference precision, typically 2-3%.
[0056] S205, perform lightweight inference transformation on the second candidate deep learning model to obtain the quantized target deep learning model.
[0057] For details regarding step S205, please refer to the above embodiments; they will not be repeated here.
[0058] In this embodiment, the target quantization accuracy of the source deep learning model is obtained, and the model quantization component is called. The model quantization component quantizes the model parameters of the first candidate deep learning model according to the target quantization accuracy to obtain the second candidate deep learning model. This application can improve the versatility of model quantization and is applicable to deep learning models of different model formats generated by multiple deep learning training frameworks. It can balance the quantization efficiency and quantization accuracy of deep learning models, reduce memory usage and parameter updates, and improve cache utilization.
[0059] Figure 3 This is a flowchart of a model quantization method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:
[0060] S301, Obtain the source deep learning model to be quantized and the first model format of the source deep learning model.
[0061] S302, obtain the second model format supported by the quantized target deep learning model, and when the first model format and the second model format are inconsistent, perform format conversion on the source deep learning model to obtain the first candidate deep learning model under the second model format.
[0062] S303, quantize the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0063] For details regarding steps S301 to S303, please refer to the above embodiments; they will not be repeated here.
[0064] S304, Listening for node build requests.
[0065] In some implementations, if the accuracy of the second candidate deep learning model meets the business requirements, there is no need to add quantization nodes and / or dequantization nodes to the specified neural network layers in the second candidate deep learning model. In other implementations, if the second candidate deep learning model obtained after quantizing the model parameters of the first candidate deep learning model based on the model quantization component does not meet the business requirements, then the node construction request is further monitored, and the construction of quantization nodes and / or dequantization nodes is performed according to the node construction request to facilitate further accuracy tuning of the second candidate deep learning model.
[0066] S305, based on the node construction request, add quantization nodes and / or dequantization nodes to the specified neural network layer in the second candidate deep learning model.
[0067] In this embodiment of the application, the specified neural network layer can be a Dense layer and / or an activation layer.
[0068] S306, Quantize and fine-tune the model parameters of the second candidate deep learning model based on quantization nodes and / or dequantization nodes.
[0069] In some implementations, the first quantization parameter of the quantization node is obtained, and the model parameters of the second candidate deep learning model are quantized based on the first quantization parameter to obtain the second candidate deep learning model with reduced bit width.
[0070] In some implementations, the second quantization parameter of the dequantization node is obtained, and the model parameters of the second candidate deep learning model are dequantized based on the second quantization parameter to obtain the second candidate deep learning model with expanded bit width.
[0071] It should be noted that, in this embodiment of the application, after quantizing and optimizing the model parameters of the second candidate deep learning model based on the quantization node and the dequantization node, the accuracy of the model parameters of the second candidate deep learning model is improved, while the target data bit width of the second candidate deep learning model remains unchanged.
[0072] S307, perform lightweight inference transformation on the second candidate deep learning model to obtain the quantized target deep learning model.
[0073] For details regarding step S37, please refer to the above embodiments; they will not be repeated here.
[0074] In this embodiment, a node construction request is monitored. Based on the request, quantization nodes and / or dequantization nodes are added to a specified neural network layer in the second candidate deep learning model. The model parameters of the second candidate deep learning model are then quantized and optimized based on the quantization nodes and / or dequantization nodes. This application improves the versatility of model quantization, is applicable to deep learning models of different format generated by multiple deep learning training frameworks, and can balance the quantization efficiency and accuracy of deep learning models, further improving the precision and accuracy of deep learning models.
[0075] Figure 4 This is a flowchart of a model quantization method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:
[0076] S401, Obtain the initial data to be processed, and construct the target inference request based on the initial data. The initial data is image data and / or voice data and / or text data.
[0077] In this embodiment, an initial inference request is received, and the initial data in the initial inference request is preprocessed to obtain a target inference request that conforms to the standard input of a symbolic mathematics system. That is, the initial data is parsed to obtain candidate data, and further, an inference request template is obtained. Information required for system processing is extracted from the candidate data, and then the filling positions of the inference request template are filled to obtain the target inference request.
[0078] In some implementations, multiple requests can be merged and batched to improve service throughput.
[0079] S402, the target inference request is input into the target deep learning model for processing to obtain the target quantization data corresponding to the initial data.
[0080] In some implementations, to improve service throughput, a wait queue, i.e., a batch of target inference requests, can be input into the target deep learning model for processing to obtain the target quantized data corresponding to the initial data. Optionally, if the model requires a Graphics Processing Unit (GPU), the target GPU can be determined from multiple candidate GPUs. For example, a candidate GPU can be randomly selected, or the GPU with the least workload can be selected from multiple candidate GPUs based on its workload. The availability of the candidate GPU is checked; if it is available, it is determined as the target GPU; otherwise, a preset candidate GPU is determined as the target GPU. Further, the target deep learning model and the target GPU are associated, device driver resources are allocated to the target GPU, and the batch of target inference requests are sent to the target GPU. The target GPU is then invoked to process the target inference requests based on the target deep learning model to obtain the target quantized data corresponding to the initial data.
[0081] In this embodiment, initial data to be processed is obtained, and a target inference request is constructed based on the initial data. The target inference request is then input into the target deep learning model for processing to obtain the target quantized data corresponding to the initial data. This application can improve the versatility of model quantization, and is applicable to deep learning models of different model formats generated by multiple deep learning training frameworks. It can balance the quantization efficiency and accuracy of deep learning models, reduce memory usage and parameter updates, improve cache utilization, and further improve the accuracy and precision of deep learning models.
[0082] like Figure 5As shown in this embodiment, a source deep learning model to be quantized is obtained, and it is determined whether the model format of the source deep learning model is TensorFlow format. If the model format of the source deep learning model is inconsistent with the symbolic mathematics system model format, the source deep learning model is converted to ONNX format to obtain a first candidate deep learning model in TensorFlow format. If the model format of the source deep learning model is consistent with the symbolic mathematics system model format, the source deep learning model is determined as the first candidate deep learning model. Further, it is determined whether the current business scenario requires high-precision quantization. In response to the current business scenario requiring high-precision quantization, the model quantization component performs QAT quantization on the model parameters of the first candidate deep learning model to obtain a second candidate deep learning model. In response to the current business scenario not requiring high-precision quantization, the model quantization component performs PTQ quantization on the model parameters of the first candidate deep learning model to obtain a second candidate deep learning model. Furthermore, the system listens for node construction requests. If a node construction request is detected, quantization nodes and / or dequantization nodes are added to a specified neural network layer in the second candidate deep learning model according to the request. The model parameters of the second candidate deep learning model are then quantized and tuned based on the quantization nodes and / or dequantization nodes. Finally, the second candidate deep learning model is subjected to TFlite transformation to obtain the quantized target deep learning model.
[0083] like Figure 6 As shown, after obtaining the quantized target deep learning model, prediction services can be provided based on the target deep learning model. That is, by integrating TFLite into the existing prediction service, prediction support for the quantized model is achieved, enabling external prediction services for the quantized model. The main process of the prediction service is as follows: First, obtain the initial data to be processed; parse the initial data to obtain candidate data; obtain the inference request template and fill it with information from the candidate data to obtain the target inference request; determine the target graphics processor from multiple candidate graphics processors and associate the target deep learning model with the target graphics processor; allocate device driver resources to the target graphics processor and send the target inference requests in batches to the target graphics processor; call the target graphics processor to process the target inference request based on the target deep learning model to obtain the target quantized data corresponding to the initial data.
[0084] This application can improve the versatility of model quantization, and is applicable to deep learning models of different model formats generated by multiple deep learning training frameworks. It can balance the quantization efficiency and quantization accuracy of deep learning models, reduce memory usage and parameter updates, and improve cache utilization.
[0085] like Figure 7As shown, based on the same concept, this application also provides a model quantization device 700, including:
[0086] The first acquisition module 710 is used to acquire the source deep learning model to be quantized and the first model format of the source deep learning model;
[0087] The second acquisition module 720 is used to acquire the second model format supported by the quantized target deep learning model, and when the first model format and the second model format are inconsistent, to perform format conversion on the source deep learning model in order to obtain the first candidate deep learning model under the second model format.
[0088] The quantization module 730 is used to quantize the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0089] The conversion module 740 is used to perform lightweight inference conversion on the second candidate deep learning model to obtain the quantized target deep learning model.
[0090] In some implementations, the quantization module 730 is also used for:
[0091] Obtain the target quantization accuracy of the source deep learning model;
[0092] The model quantization component is invoked, which quantizes the model parameters of the first candidate deep learning model according to the target quantization accuracy to obtain the second candidate deep learning model.
[0093] In some implementations, the quantization module 730 is also used for:
[0094] In response to a target quantization precision greater than or equal to a preset quantization precision threshold, the model quantization component performs training-aware quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0095] In some implementations, the quantization module 730 is also used for:
[0096] In response to the target quantization accuracy being less than the preset quantization accuracy threshold, the model quantization component performs post-training quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
[0097] In some implementations, the second acquisition module 720 is also used for:
[0098] Open neural network exchange is performed on the source deep learning model to obtain a third candidate deep learning model in the standard model format;
[0099] The third candidate deep learning model is converted from the standard model format to the second model format to obtain the first candidate deep learning model.
[0100] In some implementations, the quantization module 730 is also used for:
[0101] Listen for node build requests;
[0102] Based on the node construction request, add quantization nodes and / or dequantization nodes to the specified neural network layer in the second candidate deep learning model;
[0103] The model parameters of the second candidate deep learning model are quantized and tuned based on quantization nodes and / or dequantization nodes.
[0104] In some implementations, the quantization module 730 is also used for:
[0105] Obtain the first quantization parameter of the quantization node, and perform quantization operation on the model parameters of the second candidate deep learning model based on the first quantization parameter to obtain the second candidate deep learning model with reduced bit width; and / or
[0106] Obtain the second quantization parameter of the dequantization node, and perform dequantization operation on the model parameters of the second candidate deep learning model based on the second quantization parameter to obtain the second candidate deep learning model with expanded bit width.
[0107] In some implementations, the device also includes a processing module 750 for:
[0108] Obtain the initial data to be processed, and construct the target inference request based on the initial data, which is image data and / or voice data and / or text data;
[0109] The target inference request is input into the target deep learning model for processing to obtain the target quantized data corresponding to the initial data.
[0110] In some implementations, processing module 750 is also used for:
[0111] The initial data is parsed to obtain candidate data;
[0112] Obtain the inference request template and populate it with candidate data to obtain the target inference request.
[0113] In some implementations, processing module 750 is also used for:
[0114] The target graphics processor is determined from multiple candidate graphics processors, and the target deep learning model is associated with the target graphics processor;
[0115] Allocate device driver resources to the target graphics processor and send the target inference request to the target graphics processor;
[0116] The target graphics processor is invoked to process the target inference request based on the target deep learning model in order to obtain the target quantized data corresponding to the initial data.
[0117] This application can improve the versatility of model quantization, and is applicable to deep learning models of different model formats generated by multiple deep learning training frameworks. It can balance the quantization efficiency and quantization accuracy of deep learning models, reduce memory usage and parameter updates, and improve cache utilization.
[0118] Based on the same concept, embodiments of this application also provide an electronic device.
[0119] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 includes a memory 810, a processor 820, and a computer program product stored in the memory 810 and capable of running on the processor 820. When the processor executes the computer program, it implements the aforementioned model quantization method.
[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0124] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing computer instructions thereon, wherein the computer instructions are used to cause a computer to execute the model quantization method in the above embodiments.
[0125] Based on the same concept, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, provides the model quantization method described in the above embodiments.
[0126] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0128] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0129] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.
Claims
1. A model quantization method applied to deep learning models, characterized in that, include: Obtain the source deep learning model to be quantized and the first model format of the source deep learning model, wherein the source deep learning model includes at least one of the following: a model for image recognition, a model for image classification, and a model for speech recognition; Obtain the second model format supported by the quantized target deep learning model, and when the first model format is inconsistent with the second model format, perform format conversion on the source deep learning model to obtain the first candidate deep learning model under the second model format; The model parameters of the first candidate deep learning model are quantized to obtain the second candidate deep learning model. The second candidate deep learning model is subjected to lightweight inference transformation to obtain the quantized target deep learning model; Before performing lightweight inference transformation on the second candidate deep learning model, the method further includes: Listen for node build requests; Based on the node construction request, add quantization nodes and / or dequantization nodes to a specified neural network layer in the second candidate deep learning model; The model parameters of the second candidate deep learning model are quantized and tuned based on the quantization node and / or dequantization node. The process of data processing based on the target deep learning model includes: Acquire initial data to be processed, and construct a target inference request based on the initial data, wherein the initial data is image data and / or voice data and / or text data; The target inference request is input into the target deep learning model for processing to obtain the target quantized data corresponding to the initial data.
2. The method according to claim 1, characterized in that, The step of quantizing the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model includes: Obtain the target quantization accuracy of the source deep learning model; The model quantization component is invoked, and the model quantization component quantizes the model parameters of the first candidate deep learning model according to the target quantization accuracy to obtain the second candidate deep learning model.
3. The method according to claim 2, characterized in that, The invocation of the model quantization component, wherein the model quantization component quantizes the model parameters of the first candidate deep learning model according to the target quantization precision to obtain the second candidate deep learning model, includes: In response to the target quantization precision being greater than or equal to a preset quantization precision threshold, the model quantization component performs training-aware quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
4. The method according to claim 2, characterized in that, The invocation of the model quantization component, wherein the model quantization component quantizes the model parameters of the first candidate deep learning model according to the target quantization precision to obtain the second candidate deep learning model, includes: In response to the target quantization accuracy being less than a preset quantization accuracy threshold, the model quantization component performs post-training quantization on the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model.
5. The method according to any one of claims 1-4, characterized in that, The step of converting the source deep learning model to obtain a first candidate deep learning model in the second model format includes: The source deep learning model is subjected to open neural network exchange to obtain a third candidate deep learning model in the standard model format. The third candidate deep learning model is converted from the standard model format to the second model format to obtain the first candidate deep learning model.
6. The method according to claim 1, characterized in that, The step of quantizing and fine-tuning the model parameters of the second candidate deep learning model based on the quantization node and / or dequantization node includes: Obtain the first quantization parameter of the quantization node, and perform quantization operation on the model parameters of the second candidate deep learning model based on the first quantization parameter to obtain the second candidate deep learning model with reduced bit width; and / or The second quantization parameter of the dequantization node is obtained, and the model parameters of the second candidate deep learning model are dequantized based on the second quantization parameter to obtain the second candidate deep learning model with expanded bit width.
7. The method according to claim 1, characterized in that, The construction of the target inference request based on the initial data includes: The initial data is parsed to obtain candidate data; Obtain the inference request template and fill it with the candidate data to obtain the target inference request.
8. The method according to claim 1, characterized in that, The step of inputting the target inference request into a preset target deep learning model for processing to obtain the target quantized data corresponding to the initial data includes: A target graphics processor is determined from multiple candidate graphics processors, and the target deep learning model is associated with the target graphics processor; Allocate device driver resources to the target graphics processor and send the target inference request to the target graphics processor; The target graphics processor is invoked to process the target inference request based on the target deep learning model in order to obtain the target quantized data corresponding to the initial data.
9. A model quantization device, applied to a deep learning model, characterized in that, include: The first acquisition module is used to acquire the source deep learning model to be quantized and the first model format of the source deep learning model, wherein the source deep learning model includes at least one of the following: a model for image recognition, a model for image classification, and a model for speech recognition. The second acquisition module is used to acquire the second model format supported by the quantized target deep learning model, and when the first model format is inconsistent with the second model format, to perform format conversion on the source deep learning model to acquire the first candidate deep learning model under the second model format. The quantization module is used to quantize the model parameters of the first candidate deep learning model to obtain the second candidate deep learning model. The conversion module is used to perform lightweight inference conversion on the second candidate deep learning model to obtain the quantized target deep learning model; The quantization module is also used to listen for node construction requests; Based on the node construction request, add quantization nodes and / or dequantization nodes to a specified neural network layer in the second candidate deep learning model; The model parameters of the second candidate deep learning model are quantized and tuned based on the quantization node and / or dequantization node. The device further includes a processing module, which is used to acquire initial data to be processed and construct a target inference request based on the initial data, wherein the initial data is image data and / or voice data and / or text data; The target inference request is input into the target deep learning model for processing to obtain the target quantized data corresponding to the initial data.
10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.
12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.