A neural network model quantization deployment method supporting multiple hardware platforms

Abstract

The present invention proposes a neural network model quantization deployment method that supports multiple hardware platforms, including obtaining a neural network model to be quantized; configuring network quantization parameters for the hardware platform to be deployed; pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed; uniformly distributing and adjusting the parameters of the pre-quantized neural network model using the idea of ​​combining cross-layer weight balancing with bias absorption to obtain an adjusted neural network model; finding the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, calculating and deriving the quantized neural network model and the quantized neural network model parameters according to the best balance point; and writing the quantized neural network model parameters to the hardware platform to be deployed. The method proposed by the present invention is used to improve the inference speed of the model on the terminal side, so that the model can be deployed in computing resource-constrained devices or delay-sensitive applications.

Description

Technical Field

[0001] The present invention belongs to the field of model deployment. Background Art

[0002] In recent years, with the application of artificial intelligence in the fields of industry, cities, and medical treatment, typical open environments such as smart grids and smart cities have put forward higher requirements for deep learning theory and technology. The open environment has many scenarios, difficult tasks, high data acquisition costs, and weak cross-scenario adaptability, which brings many challenges. Therefore, it is crucial to reduce computing power costs, improve the online reasoning ability of models, and promote the application of deep learning technology in multiple industry scenarios in an open environment.

[0003] Quantization is a common method of compressing neural network models. It uses low bits (fewer bits) to represent weights and activations to reduce memory usage and accelerate the inference process. However, the additional noise it causes will lead to a decrease in model accuracy. To solve this problem, a variety of methods have been proposed to reduce the decrease in model accuracy after quantization. However, these methods usually only use general analog quantization processes to implement them, without considering the hardware platform where the model is actually deployed and the underlying implementation principles of the corresponding inference framework, resulting in serious loss of model accuracy after deployment and failure to truly implement them. Therefore, how to organically combine the actually deployed hardware inference framework and the quantization algorithm to provide a convenient and efficient low-bit quantization method for neural network models and achieve optimization and acceleration combining software and hardware has great research significance and practical value.

[0004] The main quantization methods currently include post-training quantization (PTQ) and quantization-aware training (QAT). Post-training quantization is decoupled from the training process, and is simple to operate, time-saving, and requires less calibration data, but the accuracy of the model is often seriously lost after quantization. Quantization-aware training requires training the network during the quantization process so that the network parameters can better adapt to the information loss caused by quantization. Therefore, it can better solve the problem of accuracy loss in model quantization, but it requires large-scale training data, is time-consuming, and complex to operate. At present, mainstream hardware platforms only integrate basic quantization algorithms for their respective inference frameworks, and cannot guarantee the accuracy of the model after quantization. In addition, existing quantization schemes usually only choose one of post-training quantization and quantization-aware training for quantization, and do not implement an automated quantization process. When post-training quantization cannot meet the accuracy requirements, quantization-aware training needs to be rebuilt, resulting in a complex quantization process and low efficiency. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the purpose of the present invention is to propose a neural network model quantization deployment method that supports multiple hardware platforms, so as to improve the inference speed of the model on the terminal side, so that the model can be deployed on computing resource-constrained devices or delay-sensitive applications.

[0007] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a neural network model quantization deployment method supporting multiple hardware platforms, including:

[0008] Obtain the neural network model to be quantized;

[0009] Configuring network quantization parameters for a hardware platform to be deployed, wherein the hardware platform to be deployed includes a hardware platform in an open power environment;

[0010] Pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed;

[0011] The pre-quantized neural network model parameters are uniformly adjusted using the idea of ​​combining cross-layer weight balancing with deviation absorption to obtain an adjusted neural network model;

[0012] Find the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculate and derive the quantized neural network model and quantized neural network model parameters according to the best balance;

[0013] The quantized neural network model parameters are written to the hardware platform to be deployed.

[0014] In addition, the neural network model quantization deployment method supporting multiple hardware platforms according to the above embodiment of the present invention may also have the following additional technical features:

[0015] Furthermore, in one embodiment of the present invention, configuring network quantization parameters for the hardware platform to be deployed includes:

[0016] Convert the neural network model to be quantized into an intermediate format supported by the hardware platform to be deployed for reading;

[0017] The network quantization parameters include the step size scale and the zero point zero_point, which are used to describe the graph fusion logic, scheduling scheme, quantization detail strategy and high-order quantization algorithm of the underlying hardware.

[0018] Further, in one embodiment of the present invention, the pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed includes:

[0019] Performing low-bit quantization on the neural network model to be quantized according to the quantization strategy integrated in the inference framework of the hardware platform to be deployed;

[0020] Determine that the accuracy loss of the neural network model to be quantized after low-bit quantization is within an acceptable error range;

[0021] If the acceleration ratio of the neural network model to be quantized after the low-bit quantization meets the preset value, deriving the pre-quantized neural network model and pre-quantized neural network model parameters;

[0022] On the contrary, the baseline size of the control group is adjusted to maximize the GPU utilization, and then the acceleration ratio of the neural network model to be quantized after adjusting the baseline size is determined to be in line with the preset value.

[0023] Further, in one embodiment of the present invention, the idea of ​​combining cross-layer weight balancing with deviation absorption is used to uniformly distribute the parameters of the pre-quantized neural network model, including:

[0024] According to formula (1), the scaling coefficients of different channels between the upper and lower layers are adjusted.

[0025]

[0026] in, S is a diagonal matrix, and each value on the diagonal is a factor used to adjust the scaling factor;

[0027] According to formula (2), the quantization error is reduced.

[0028]

[0029] in, c is a constant.

[0030] Further, in one embodiment of the present invention, the method of finding the optimal balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculating and deriving the quantized neural network model and the quantized neural network model parameters according to the optimal balance point includes:

[0031] Parsing the adjusted neural network model into a quantized computational graph;

[0032] According to the quantization operator types supported by the hardware platform to be deployed, operators that need to be quantized and operators that do not need to be quantized are divided from the quantization calculation graph;

[0033] Reading configuration information of a quantization parameter configuration module, and initializing quantization information for the operator to be quantized;

[0034] The neural network model after the initialization operator is post-trained and quantized. If the quantization error of the neural network model after the initialization operator meets the preset value, the quantization is successful.

[0035] Further, in one embodiment of the present invention, if the quantization error of the neural network model after the initialization operator does not meet the preset value, optimization is performed through an adaptive graph scheduling algorithm, including:

[0036] Obtain the quantization error of each layer of the neural network model after the initialization operator by a layer-by-layer error analysis method, and obtain the layers whose quantization errors are greater than the preset value to form a scheduling set S;

[0037] Traversing the scheduling set S, performing graph scheduling optimization, and scheduling each layer in the scheduling set S to a high-precision platform for quantization;

[0038] If the quantization error of the neural network model optimized by graph scheduling meets the preset value, the quantization is successful.

[0039] Further, in one embodiment of the present invention, if the quantization error of the neural network model after graph scheduling optimization does not meet the preset value, quantization-aware training is performed on the neural network model after graph scheduling optimization, including:

[0040] Inserting pseudo-quantization nodes into the neural network model optimized by graph scheduling, and learning the quantization step of weights and activation values ​​layer by layer in the neural network of the neural network model optimized by graph scheduling;

[0041] If the quantization error of the neural network model after quantization perception training meets the preset value, the quantization is successful.

[0042] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a neural network model quantization deployment device supporting multiple hardware platforms, including the following modules:

[0043] An acquisition module, used to acquire the neural network model to be quantized;

[0044] A quantization parameter configuration module, used to configure network quantization parameters for a hardware platform to be deployed, wherein the hardware platform to be deployed includes a hardware platform in an open power environment;

[0045] A prequantization module, used to prequantize the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed, and derive the prequantized neural network model and prequantized neural network model parameters;

[0046] A model distribution adjustment module, used to uniformly adjust the parameters of the pre-quantized neural network model by combining cross-layer weight balancing with deviation absorption to obtain an adjusted neural network model;

[0047] A quantization engine module, used to find the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculate and derive the quantized neural network model and quantized neural network model parameters according to the best balance point;

[0048] The model deployment module is used to write the quantized neural network model parameters to the hardware platform to be deployed.

[0049] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the above-mentioned method for quantizing and deploying a neural network model that supports multiple hardware platforms.

[0050] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, a method for quantizing and deploying a neural network model that supports multiple hardware platforms as described above is implemented.

[0051] The embodiment of the present invention proposes a neural network model quantization deployment method that supports multiple hardware platforms. It combines the quantization strategies and graph fusion strategies within different hardware platforms, uses high-order quantization algorithms such as AdaRound and CLE (Cross-LayerEqualization), adjusts the model parameter distribution before quantization to make it more suitable for quantization, combines the actual optimization acceleration process of the inference framework during quantization, selects appropriate quantization calculation rules and network quantization rules, analyzes the quantization error layer by layer after quantization, and uses an adaptive graph scheduling strategy to find the best balance between the accuracy and speed of the model after quantization, thereby realizing quantization optimization acceleration combined with software and hardware, thereby providing a highly available end-to-end neural network model quantization compression service for the end-side platform with limited computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A flowchart of a neural network model quantization deployment method supporting multiple hardware platforms provided by an embodiment of the present invention.

[0054] Figure 2A flow chart of a model pre-quantization module provided by an embodiment of the present invention.

[0055] Figure 3 A processing flow chart of a quantization engine module provided in an embodiment of the present invention.

[0056] Figure 4 A schematic diagram of a process flow of a neural network model quantization deployment device supporting multiple hardware platforms provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0058] The following describes a neural network model quantization deployment method supporting multiple hardware platforms according to an embodiment of the present invention with reference to the accompanying drawings.

[0059] Figure 1 A flowchart of a neural network model quantization deployment method supporting multiple hardware platforms provided by an embodiment of the present invention.

[0060] like Figure 1 As shown, the neural network model quantization deployment method supporting multiple hardware platforms includes the following steps:

[0061] S101: Obtain a neural network model to be quantized;

[0062] Specifically, obtain the trained neural network model to be quantized from the outside, and convert it into an intermediate format supported by mainstream hardware platforms (such as ONNX files) for reading, so as to facilitate subsequent conversion to the target format of the inference framework of different hardware platforms; obtain a calibration data set that can effectively represent the actual input situation of the model; obtain the training data set used in the model pre-training stage for subsequent quantization-aware training. If the training data set is not provided, use the calibration data set for quantization-aware training; obtain a model verification data set for calculation and comparison of model accuracy before and after quantization.

[0063] S102: configuring network quantitative parameters for a hardware platform to be deployed, where the hardware platform to be deployed includes a hardware platform in an open power environment;

[0064] Furthermore, in one embodiment of the present invention, configuring network quantization parameters for the hardware platform to be deployed includes:

[0065] Convert the neural network model to be quantized into an intermediate format supported by the hardware platform to be deployed for reading;

[0066] Among them, the network quantization parameters include step size scale and zero point zero_point, which are used to describe the graph fusion logic, scheduling scheme, quantization detail strategy and high-order quantization algorithm of the underlying hardware.

[0067] Specifically, determine the hardware platform to be deployed and configure the network quantization parameters according to the inference framework it provides to control the quantization logic of the quantization engine module. It mainly describes the graph fusion logic, scheduling scheme, quantization detail strategy and high-order quantization algorithm use of the underlying hardware to ensure that the error between the quantization simulator and the actual hardware is small.

[0068] S103: Pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed;

[0069] Further, in one embodiment of the present invention, pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed includes:

[0070] Perform low-bit quantization on the neural network model to be quantized based on the quantization strategy integrated in the inference framework of the hardware platform to be deployed;

[0071] Determine that the accuracy loss of the neural network model to be quantized after low-bit quantization is within an acceptable error range;

[0072] If the acceleration ratio of the neural network model to be quantized after low-bit quantization meets the preset value, the pre-quantized neural network model and pre-quantized neural network model parameters are derived;

[0073] On the contrary, the baseline size of the control group is adjusted to maximize the GPU utilization, and then the acceleration ratio of the neural network model to be quantized after adjusting the baseline size is determined to be in line with the preset value.

[0074] Specifically, Figure 2 As shown, the model is pre-quantized using the benchmark quantization method. The steps are as follows:

[0075] 1) Perform low-bit quantization based on the quantization strategy integrated into the selected hardware platform inference framework;

[0076] 2) Determine whether the accuracy loss of the model after quantization is within the acceptable error range. If so, go to step 3). Otherwise, it means that the benchmark quantization has failed and the basic quantization operation of the hardware does not meet the accuracy requirements. It is necessary to execute the subsequent modules of this solution to quantize.

[0077] 3) Evaluate whether the model acceleration ratio meets expectations. If so, go to step 5), otherwise go to step 4).

[0078] 4) Adjust the batch size to maximize GPU utilization, and then go to step 3) for judgment.

[0079] 5) Export the quantized model file and quantization parameter file, and the entire quantization process is completed;

[0080] 6) Execute the final model deployment module.

[0081] S104: uniformly distribute and adjust the parameters of the pre-quantized neural network model by combining cross-layer weight balancing and deviation absorption to obtain an adjusted neural network model;

[0082] Furthermore, in one embodiment of the present invention, the idea of ​​combining cross-layer weight balancing with deviation absorption is used to uniformly distribute and adjust the parameters of the pre-quantized neural network model, including:

[0083] According to formula (1), the scaling coefficients of different channels between the upper and lower layers are adjusted.

[0084]

[0085] in, S is a diagonal matrix, and each value on the diagonal is a factor used to adjust the scaling factor;

[0086] According to formula (2), the quantization error is reduced.

[0087]

[0088] in, c is a constant.

[0089] Specifically, the present invention combines the idea of ​​cross-layer weight equalization (CLE) with bias absorption to make the model parameters closer to uniform distribution. Compared with normal distribution, uniform distribution is more robust to changes in quantization parameters. The calculation formula is shown in (1):

[0090]

[0091] in, S is a diagonal matrix, and each value on the diagonal is a factor used to adjust the scaling factor. Therefore, the scaling characteristics of the piecewise linear function can be used: f(sx)=sf(x) to adjust the scaling factors of different channels between the upper and lower layers.

[0092] At the same time, when the scaling factor of a channel of a layer weight is greater than 1, the data range of the activation value of the channel will increase, which may cause the data range difference of the activation value between channels to become larger, resulting in a larger quantization error. Therefore, it is necessary to directly subtract a constant c from the activation value of the layer to reduce the entire data range, thereby minimizing the quantization error. The bias value of the next layer will absorb the subtracted constant c. The calculation formula is shown in (2):

[0093]

[0094] in, c is a constant.

[0095] S105: finding the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculating and deriving the quantized neural network model and the parameters of the quantized neural network model according to the best balance;

[0096] Further, in one embodiment of the present invention, an optimal balance between the accuracy and speed of the adjusted neural network model is found using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and a quantized neural network model and quantized neural network model parameters are calculated and derived according to the optimal balance point, including:

[0097] Parse the adjusted neural network model into a quantized computational graph;

[0098] According to the types of quantized operators supported by the hardware platform to be deployed, operators that need to be quantized and operators that do not need to be quantized are divided from the quantized calculation graph;

[0099] Read the configuration information of the quantization parameter configuration module and initialize the quantization information for the operator that needs to be quantized;

[0100] The neural network model after the initialization operator is post-trained and quantized. If the quantization error of the neural network model after the initialization operator meets the preset value, the quantization is successful.

[0101] Furthermore, in one embodiment of the present invention, if the quantization error of the neural network model after the initialization operator does not meet the preset value, optimization is performed through an adaptive graph scheduling algorithm, including:

[0102] The quantization error of each layer of the neural network model after the initialization operator is obtained by a layer-by-layer error analysis method, and the layers whose quantization errors are greater than the preset value are obtained to form a scheduling set S;

[0103] Traverse the scheduling set S, perform graph scheduling optimization, and schedule each layer in the scheduling set S to the high-precision platform for quantization;

[0104] If the quantization error of the neural network model optimized by graph scheduling meets the preset value, the quantization is successful.

[0105] Further, in one embodiment of the present invention, if the quantization error of the neural network model after graph scheduling optimization does not meet the preset value, the neural network model after graph scheduling optimization is subjected to quantization-aware training, including:

[0106] Insert pseudo-quantization nodes into the neural network model optimized by graph scheduling, and learn the quantization step of weights and activation values ​​layer by layer in the neural network of the neural network model optimized by graph scheduling;

[0107] If the quantization error of the neural network model trained with quantization awareness meets the preset value, the quantization is successful.

[0108] Specifically, the present invention proposes an adaptive graph scheduling algorithm, which obtains the quantization error of each layer through a layer-by-layer error analysis method, and gives greater weights to nodes close to the network input layer and nodes with serious local cumulative errors in the network graph, so that they can be preferentially scheduled to execute on a hardware platform with higher precision, thereby achieving a balance between the accuracy and performance of the quantized model. The following is a detailed introduction to its implementation details.

[0109] After the network completes the quantization operation, first unquantize all operators in the network, and then quantize each computational operator separately, and use the error calculation method of quantization signal-to-noise ratio (SNR) to measure the impact of a single operator on the network output. The calculation method of quantization error is shown in the following formula.

[0110]

[0111] Among them, X represents the original floating-point value before quantization, Q(X) is the output value after quantization, and N is the tensor length.

[0112] After obtaining the output error of each layer node, initialize its scheduling weight according to the ascending order of the error value. For layers with the same quantization error, calculate the path distance d from the node to the network output layer. i, and its product with the distance scaling factor σ is added to the initial scheduling weight as a weighted value. For continuous nodes in the network graph with serious local cumulative errors, the product of the number of nodes and the cumulative scaling factor τ is calculated and weighted with the scheduling weight. At the same time, combined with the operator fusion strategy at the bottom of the hardware, the internal operator scheduling weights that conform to the operator fusion mode are processed in a consistent manner to ensure that the operator fusion is not affected by the graph scheduling, thereby making full use of registers and caches, reducing the time spent on storing and reading data between the GPU and the video memory, and improving the inference performance. Finally, the scheduling weights of each layer are sorted in descending order to obtain the priority scheduling sequence of the operator. Users can select operators from the priority scheduling queue in order to schedule to the FP32-bit platform according to actual application requirements. In this way, the inference speed can be maximized while improving the accuracy of the model after quantization, realizing the quantization optimization acceleration combined with software and hardware. The complete algorithm details are shown in Table 1.

[0113] Table 1

[0114]

[0115] The specific processing flow in the present invention is as follows Figure 3 As shown, the steps are as follows:

[0116] 1) First, parse the neural network model in Onnx, Pytorch, Tensorflow, Caffe and other formats into a quantized calculation graph;

[0117] 2) According to the types of quantization operators supported by the selected hardware platform, operators that need to be quantized and operators that do not need to be quantized are separated from the computation graph;

[0118] 3) Read the configuration information of the quantization parameter configuration module, initialize the quantization information for all operators that need to be quantized, and establish a quantization pipeline;

[0119] 4) Perform post-training quantization, use the quantization strategy of adaptive task loss to round, and convert the optimization of weights into the optimization of the results after convolution calculation. The calculation formula is shown in (3):

[0120]

[0121] Among them, f a () is the activation function. is the dequantized input of the current layer, and x is the floating-point input of the current layer;

[0122] 5) Determine whether the quantization error meets the requirements. If so, the quantization is successful and the process ends. Otherwise, go to step 6);

[0123] 6) Perform layer-by-layer quantization error analysis. If the layer error is greater than E t, then the quantization sensitivity of this layer is considered to be large, and it is combined into a scheduling set S;

[0124] 7) Traverse the scheduling set S, perform graph scheduling optimization, and schedule each layer in the set S to the high-precision platform for quantization;

[0125] 8) Determine whether the performance after graph scheduling meets the requirements. If yes, go to step 5), otherwise go to step 9);

[0126] 9) Perform quantization-aware training, insert pseudo-quantization nodes, and learn the quantization step of weights and activation values ​​layer by layer in the neural network to improve quantization accuracy. The main formulas are shown in (4) and (5):

[0127]

[0128]

[0129] Among them, clip(z, r1, r2) returns the value of z. If it is less than r1, it is set to r1, and if it is less than r2, it is set to r2. The round() function is responsible for rounding z to the nearest integer.

[0130] The step gradient calculation is shown in formula (6):

[0131]

[0132] The scaling of the step gradient is shown in formula (7):

[0133]

[0134] 10) Determine whether the quantization error meets the requirements, if so, go to step 11), otherwise, a higher precision quantization is required;

[0135] 11) Export the quantized model file and quantization parameter file. The quantization is successful and the process ends.

[0136] S106: Writing the quantized neural network model parameters into the hardware platform to be deployed.

[0137] The quantization parameters obtained after quantization are written into the runtime model using the tools of the inference library, so that the quantized model can be executed on the hardware. Finally, the output results of the quantization simulator are compared with the actual execution results of the hardware to perform error verification.

[0138] The embodiment of the present invention proposes a neural network model quantization deployment method that supports multiple hardware platforms. It combines the quantization strategies and graph fusion strategies within different hardware platforms, uses high-order quantization algorithms such as AdaRound and CLE (Cross-LayerEqualization), adjusts the model parameter distribution before quantization to make it more suitable for quantization, combines the actual optimization acceleration process of the inference framework during quantization, selects appropriate quantization calculation rules and network quantization rules, analyzes the quantization error layer by layer after quantization, and uses an adaptive graph scheduling strategy to find the best balance between the accuracy and speed of the model after quantization, thereby realizing quantization optimization acceleration combined with software and hardware, thereby providing a highly available end-to-end neural network model quantization compression service for the end-side platform with limited computing power.

[0139] Compared with the existing technology, firstly, the existing closest technology does not consider the optimization strategy of the reasoning framework of different hardware platforms during the quantization process, and the difference between the simulated quantization and the real hardware output is large, resulting in a serious decrease in accuracy. Second, the present invention configures quantization parameters and inserts quantization nodes for different hardware platforms during the quantization process, so that the quantization simulator can correctly simulate the hardware reasoning process, and realizes the quantization optimization acceleration of the combination of software and hardware. Secondly, most of the existing closest technologies use the quantization scheme that comes with the hardware platform reasoning library. The steps are simple, but the accuracy is seriously lost after quantization. The present invention constructs a set of automated quantization pipelines, optimizes the model distribution before quantization, realizes the dynamic combination of post-training quantization and quantization-aware training during quantization, and uses error analysis and graph scheduling after quantization, which effectively improves the accuracy of the model after quantization and the efficiency of quantization deployment.

[0140] In order to implement the above embodiments, the present invention also proposes a neural network model quantization deployment device that supports multiple hardware platforms.

[0141] Figure 4 A schematic diagram of the structure of a neural network model quantization deployment device supporting multiple hardware platforms provided in an embodiment of the present invention.

[0142] like Figure 4 As shown, the neural network model quantization deployment device supporting multiple hardware platforms includes: an acquisition module, a quantization parameter configuration module, a pre-quantization module, a model distribution adjustment module, a quantization engine module, and a model deployment module, wherein:

[0143] An acquisition module, used to acquire the neural network model to be quantized;

[0144] A quantization parameter configuration module, used to configure network quantization parameters for a hardware platform to be deployed, where the hardware platform to be deployed includes a hardware platform in an open power environment;

[0145] A prequantization module, used to prequantize the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed, and derive the prequantized neural network model and prequantized neural network model parameters;

[0146] The model distribution adjustment module is used to uniformly adjust the parameters of the pre-quantized neural network model by combining cross-layer weight balancing with bias absorption to obtain the adjusted neural network model;

[0147] A quantization engine module, used to find the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculate and derive the quantized neural network model and quantized neural network model parameters according to the best balance point;

[0148] The model deployment module is used to write the quantized neural network model parameters to the hardware platform to be deployed.

[0149] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the above-mentioned method for quantizing and deploying a neural network model that supports multiple hardware platforms.

[0150] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the method for quantizing and deploying a neural network model that supports multiple hardware platforms as described above is implemented.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0152] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0153] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for quantizing and deploying a neural network model that supports multiple hardware platforms, characterized in that: The following steps are involved: Obtain the neural network model to be quantized; Configuring network quantization parameters for a hardware platform to be deployed, wherein the hardware platform to be deployed includes a hardware platform in an open power environment; Pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed; The pre-quantized neural network model parameters are uniformly adjusted using the idea of ​​combining cross-layer weight balancing with bias absorption to obtain an adjusted neural network model; Find the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculate and derive the quantized neural network model and quantized neural network model parameters according to the best balance; Writing the quantized neural network model parameters into the hardware platform to be deployed; The idea of ​​combining cross-layer weight balancing with deviation absorption to uniformly distribute and adjust the parameters of the pre-quantized neural network model includes: According to formula (1), the scaling coefficients of different channels between the upper and lower layers are adjusted. （1） in, , , , S is a diagonal matrix, and each value on the diagonal is a factor used to adjust the scaling factor; According to formula (2), the quantization error is reduced. （2） in, , , , c is a constant.

2. The method according to claim 1, characterized in that The configuring of network quantization parameters for the hardware platform to be deployed includes: Convert the neural network model to be quantized into an intermediate format supported by the hardware platform to be deployed for reading; The network quantization parameters include the step size scale and the zero point zero_point, which are used to describe the graph fusion logic, scheduling scheme, quantization detail strategy and high-order quantization algorithm of the underlying hardware.

3. The method according to claim 1, characterized in that The pre-quantizing the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed includes: Performing low-bit quantization on the neural network model to be quantized according to the quantization strategy integrated in the inference framework of the hardware platform to be deployed; Determine that the accuracy loss of the neural network model to be quantized after low-bit quantization is within an acceptable error range; If the acceleration ratio of the neural network model to be quantized after the low-bit quantization meets the preset value, deriving the pre-quantized neural network model and pre-quantized neural network model parameters; On the contrary, the baseline size of the control group is adjusted to maximize the GPU utilization, and then the acceleration ratio of the neural network model to be quantized after adjusting the baseline size is determined to be in line with the preset value.

4. The method according to claim 1, characterized in that: The method of finding the optimal balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculating and deriving the quantized neural network model and the parameters of the quantized neural network model according to the optimal balance point includes: Parsing the adjusted neural network model into a quantized computational graph; According to the quantization operator types supported by the hardware platform to be deployed, operators that need to be quantized and operators that do not need to be quantized are divided from the quantization calculation graph; Read the configuration information of the quantization parameter configuration module, and initialize the quantization information for the operator that needs to be quantized; The neural network model after the initialization operator is post-trained and quantized. If the quantization error of the neural network model after the initialization operator meets the preset value, the quantization is successful.

5. The method according to claim 4, characterized in that If the quantization error of the neural network model after the initialization operator does not meet the preset value, optimization is performed through an adaptive graph scheduling algorithm, including: Obtain the quantization error of each layer of the neural network model after the initialization operator by a layer-by-layer error analysis method, and obtain the layers whose quantization errors are greater than the preset value to form a scheduling set S; Traversing the scheduling set S, performing graph scheduling optimization, and scheduling each layer in the scheduling set S to a high-precision platform for quantization; If the quantization error of the neural network model optimized by graph scheduling meets the preset value, the quantization is successful.

6. The method according to claim 5, characterized in that If the quantization error of the neural network model after graph scheduling optimization does not meet the preset value, quantization-aware training is performed on the neural network model after graph scheduling optimization, including: Inserting pseudo-quantization nodes into the neural network model optimized by graph scheduling, and learning the quantization step of weights and activation values ​​layer by layer in the neural network of the neural network model optimized by graph scheduling; If the quantization error of the neural network model after quantization perception training meets the preset value, the quantization is successful.

7. A neural network model quantization deployment device supporting multiple hardware platforms, characterized in that: Includes the following modules: An acquisition module, used to acquire the neural network model to be quantized; A quantization parameter configuration module, used to configure network quantization parameters for a hardware platform to be deployed, wherein the hardware platform to be deployed includes a hardware platform in an open power environment; A prequantization module, used to prequantize the neural network model to be quantized using a benchmark quantization method according to the hardware platform to be deployed, and derive the prequantized neural network model and prequantized neural network model parameters; A model distribution adjustment module, used to uniformly adjust the parameters of the pre-quantized neural network model by combining cross-layer weight balancing with deviation absorption to obtain an adjusted neural network model; A quantization engine module, used to find the best balance between the accuracy and speed of the adjusted neural network model using an adaptive graph scheduling strategy according to the hardware platform to be deployed, and calculate and derive the quantized neural network model and quantized neural network model parameters according to the best balance point; A model deployment module, used to write the quantized neural network model parameters into the hardware platform to be deployed; Wherein, the model distribution adjustment module is also used for: According to formula (1), the scaling coefficients of different channels between the upper and lower layers are adjusted. （1） in, , , , S is a diagonal matrix, and each value on the diagonal is a factor used to adjust the scaling factor; According to formula (2), the quantization error is reduced. （2） in, , , , c is a constant.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for quantizing and deploying a neural network model supporting multiple hardware platforms as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for quantizing and deploying a neural network model supporting multiple hardware platforms as described in any one of claims 1 to 6 is implemented.

Images