Model quantization method, apparatus, device, computer program, and storage medium

Abstract

The embodiment of the present disclosure discloses a model quantization method, device, equipment, computer program and storage medium, which can improve the accuracy of model quantization and the universality of the deployment of the quantized model. The method comprises: performing model inference by using an initial model to be quantized to obtain an initial data set; performing simulated quantization on the initial model based on a quantization mode of a target deployment platform to obtain a quantized data set; the simulated quantization is used for quantizing and restoring the activation value output by each network layer in the initial model and then entering the next network layer for inference; calculating a quantization error set based on the initial data set and the quantized data set; adjusting the initial model based on the quantization error set to obtain a quantized model; generating a quantization parameter based on the truncated value of the activation value after the simulated quantization of each network layer; and deploying the quantized model to the target deployment platform by using the quantization parameter.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a model quantization method, device, equipment, computer program and storage medium. BACKGROUND

[0002] At present, in the deployment and application process of neural network models such as deep learning models, model quantization is usually used to significantly reduce the size of the model, shorten the running time of the model, and improve the algorithm efficiency. However, the model will encounter many challenges in the quantization process. First, model quantization, such as quantization from floating point to 8-bit integer, often causes loss of accuracy. Related technologies usually use mixed precision methods to reduce the loss of accuracy or reduce the improvement of accuracy, which is very limited. Secondly, the current model quantization tools are for the software / hardware configuration of the target deployment platform to perform model quantization, thereby reducing the universality of the quantized model in other platforms. SUMMARY

[0003] The embodiments of the present disclosure aim to provide a model quantization method, device, equipment, computer program and storage medium, which can improve the accuracy of model quantization and the universality of the deployment of the quantized model.

[0004] The technical solution of the present disclosure is implemented as follows:

[0005] The embodiments of the present disclosure provide a model quantization method, comprising:

[0006] performing model inference using an initial model to be quantized to obtain an initial data set; the initial data set contains the truncated values of the activation values output by each network layer in the initial model;

[0007] performing simulated quantization on the initial model based on the quantization mode of a target deployment platform to obtain a quantized data set; the simulated quantization is used to quantize and restore the activation values output by each network layer in the initial model and then perform inference in the next network layer; the quantized data set contains the truncated values of the activation values after simulated quantization of each network layer;

[0008] calculating a quantization error set based on the initial data set and the quantized data set; and adjusting the initial model based on the quantization error set to obtain a quantized model;

[0009] generating quantization parameters based on the truncated values of the activation values after simulated quantization of each network layer; the quantization parameters are used to deploy the quantized model to the target deployment platform.

[0010] The embodiments of the present disclosure provide a model quantization device, comprising:

[0011] An inference module is configured to perform model inference on an initial model to be quantized to obtain an initial data set, wherein the initial data set comprises truncated values of activation values output by each network layer in the initial model;

[0012] An analog quantization module is configured to perform analog quantization on the initial model based on a quantization mode of a target deployment platform to obtain a quantization data set, wherein the analog quantization is configured to quantize and restore the activation values output by each network layer in the initial model and then perform inference on the next network layer, and the quantization data set comprises truncated values of the activation values of each network layer after analog quantization.

[0013] An error adjustment module is configured to calculate a quantization error set based on the initial data set and the quantization data set, and adjust the initial model based on the quantization error set to obtain a quantization model.

[0014] A parameter generation module is configured to generate quantization parameters based on the truncated values of the activation values of each network layer after analog quantization, wherein the quantization parameters are used to deploy the quantization model to the target deployment platform.

[0015] In the above device, the inference module is further configured to obtain quantization calibration data, perform model inference on the quantization calibration data through each network layer in the initial model to obtain at least one activation value output by each network layer, and perform statistical truncation processing on the at least one activation value of each network layer by using at least one preset statistical algorithm to obtain a truncated value of the activation value corresponding to each network layer as the initial data set.

[0016] In the above device, the truncated value of the activation value corresponding to each network layer comprises any one of the following:

[0017] The maximum value in the at least one activation value, the minimum value in the at least one activation value, or the variance value of the median of the at least one activation value.

[0018] In the device, the analog quantization module is further configured to: for each network layer in the initial model, obtain a quantization scale corresponding to the each network layer according to a truncation value of an activation value corresponding to the each network layer; infer, by using an i-th layer network in the initial model, i-1-th quantized data output by an i-1-th layer to obtain an i-th initial activation value, where i is a positive integer greater than or equal to 2, and the first quantized data is obtained by inferring, by using a first layer network in the initial model, the quantization calibration data, and performing quantization processing and scale restoration on a first initial activation value obtained by inference; perform quantization processing and scale restoration on the i-th initial activation value according to the quantization scale corresponding to the i-th layer network, in combination with the quantization mode and a preset quantization precision, to obtain i-th quantized data, so as to realize analog quantization on the i-th network layer, until analog quantization is realized on each network layer in the initial model, and the quantized data set is obtained.

[0019] In the device, the quantization mode includes a zero coefficient, and the analog quantization module is further configured to: take an integer of a ratio of the i-th initial activation value to the quantization scale to obtain an i-th initial quantization value; adjust the i-th initial quantization value by using the zero coefficient to obtain an i-th intermediate quantization value; and perform truncation on the i-th intermediate quantization value by using the preset quantization precision to obtain an i-th quantization value, so as to complete a quantization processing process; and take a product of the i-th quantization value and the quantization scale as the i-th quantized data, so as to complete a scale restoration process.

[0020] In the device, the error adjustment module is further configured to: calculate a cosine distance between each initial data in the initial data set and each quantized data in the quantized data set as a quantization error corresponding to the each network layer, so as to obtain the quantization error set.

[0021] In the device, the error adjustment module is further configured to: in a case where it is determined, by performing error numerical analysis and / or error distribution analysis on the quantization error set, that a preset adjustment condition is reached, determine an error adjustment manner according to a platform type of the target deployment platform; and adjust network parameters of the initial model by using the error adjustment manner, to obtain the quantized model.

[0022] In the device, the error adjustment module is further configured to: in a case where it is determined, by performing error numerical analysis and / or error distribution analysis on the quantization error set, that a preset adjustment condition is reached, adjust network parameters of the initial model based on an analysis result of the error numerical analysis and / or the error distribution analysis, to obtain the quantized model.

[0023] In the apparatus, the error adjustment module is further configured to, in a case where it is determined that the preset adjustment condition is met by performing error numerical analysis and / or error distribution analysis on the set of quantization errors, adjust the network parameters of the initial model by performing tensor comparison on the initial data set and the quantized data set, to obtain the quantized model.

[0024] In the apparatus, the error adjustment module is further configured to obtain tensor distribution information by comparing tensor numerical distribution of the initial data set and the quantized data set, and / or obtain quantization grouping information by comparing tensor scale information of the initial data set and the quantized data set; the tensor scale represents a grouping form used for quantization; the tensor distribution information and / or the quantization grouping information are used as error performance information, and an error adjustment manner is determined based on the error performance information or based on a combination of the error performance information and a platform type of a target deployment platform; the network parameters of the initial model are adjusted by using the error adjustment manner, to obtain the quantized model.

[0025] In the apparatus, the error adjustment module is further configured to obtain distribution feature difference of tensor numerical values of the initial data set and the quantized data set in each channel according to the tensor distribution information in the error performance information; in a case where the distribution feature difference meets a preset tensor distribution adjustment condition, determine a model cross-layer averaging algorithm as the error adjustment manner; and / or, in a case where the quantization grouping information in the error performance information represents that there are at least two quantization groupings, determine a bias correction algorithm as the error adjustment manner; and / or, in a case where the distribution feature difference meets the preset tensor distribution adjustment condition or the quantization grouping information represents that there are at least two quantization groupings, determine a preset quantization precision corresponding to the platform type; in a case where the preset quantization precision is higher than a preset precision threshold, determine an adaptive rounding algorithm as the error adjustment manner.

[0026] In the apparatus, the error adjustment module is further configured to, in a case where a maximum value of the quantization errors contained in the set of quantization errors is greater than a first preset error threshold, determine that the preset adjustment condition is met; and / or, in a case where a number of quantization errors greater than a second preset error threshold in the set of quantization errors is greater than an error number threshold, determine that the preset adjustment condition is met; the second preset error threshold is smaller than the first preset error threshold.

[0027] In the apparatus, the parameter generation module is configured to perform at least one of parameter screening, serialization, and packaging on the truncated value of the simulated quantized activation value of each network layer according to a preset quantization index of a target deployment platform, to generate the quantization parameter.

[0028] The embodiment of the present disclosure provides an electronic device, comprising:

[0029] a memory for storing executable instructions;

[0030] a processor for executing the executable instructions stored in the memory to implement the model quantization method provided by the embodiment of the present disclosure.

[0031] The embodiment of the present disclosure provides a computer-readable storage medium storing executable instructions for causing a processor to execute when the model quantization method provided by the embodiment of the present disclosure is implemented.

[0032] The embodiment of the present disclosure provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the model quantization method provided by the embodiment of the present disclosure.

[0033] The embodiment of the present disclosure has the following beneficial effects:

[0034] The initial data set and the quantized data set are compared for errors by analog quantization to obtain a quantization error set; thus, error analysis can be performed based on the quantization error set, and the initial model can be adjusted, thereby effectively improving the precision of the quantized model. Moreover, based on the quantization mode of the target deployment platform, the initial model is analog quantized to generate quantized parameters for deployment on the target deployment platform, supporting multi-platform deployment, and different quantization modes can be used for model quantization according to different hardware and software configurations of the target deployment platform to generate different quantization parameters. Thus, the quantized model can be deployed on the target deployment platform using the quantization parameters corresponding to the platform, thereby realizing high-precision, multi-platform model quantization and improving the universality of the quantized model application. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is an optional flowchart of the model quantization method provided by the embodiment of the present disclosure;

[0036] Figure 2 is an optional flowchart of the model quantization method provided by the embodiment of the present disclosure;

[0037] Figure 3 is an optional flowchart of the model quantization method provided by the embodiment of the present disclosure;

[0038] Figure 4 is an optional flowchart of the model quantization method provided by the embodiment of the present disclosure;

[0039] Figure 5 is an optional flowchart of the model quantization method provided by the embodiment of the present disclosure;

[0040] Figure 6 is an optional flow diagram of a model quantization method provided by an embodiment of the present disclosure;

[0041] Figure 7 is an optional flow diagram of a model quantization method applied to an actual scenario provided by an embodiment of the present disclosure;

[0042] Figure 8 is a structural diagram of a model quantization device provided by an embodiment of the present disclosure;

[0043] Figure 9 is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present disclosure, and all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure.

[0045] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0046] In the following description, the terms “first\second\third” are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that “first\second\third” can be interchanged in a specific order or sequence as allowed, so that the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0047] The term “and / or” herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term “at least one” herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0049] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure.

[0050] The model quantization method, device, equipment, computer program and storage medium provided in the embodiments of the present disclosure can improve the precision of model quantization and the universality of the deployment of the quantized model. The following describes an exemplary application of the electronic device provided in the embodiments of the present disclosure. The electronic device provided in the embodiments of the present disclosure can be implemented as a variety of types of terminals or user terminals such as a smartphone, a smart watch, a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), a smart voice interaction device, a smart home appliance and a vehicle terminal, and can also be implemented as a server. As another implementable manner of the embodiments of the present disclosure, the server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms, but is not limited thereto.

[0051] The following describes a model quantization method of the embodiments of the present disclosure, taking an electronic device as an execution subject. Figure 1 is an optional flowchart of the trajectory management method provided in the embodiments of the present disclosure, which will be described in combination with the steps shown in Figure 1 .

[0052] S101, model inference is performed by using an initial model to be quantized to obtain an initial data set; the initial data set contains a truncated value of an activation value output by each network layer in the initial model.

[0053] The embodiments of the present disclosure are applicable to quantizing a trained neural network model to obtain a quantized model with reduced model volume, so as to deploy the quantized model to an application platform and implement related functions of the neural network model on the application platform.

[0054] In the embodiments of the present disclosure, an electronic device obtains a neural network model with original precision that has completed training as an initial model to quantize the initial model.

[0055] In the embodiments of the present disclosure, an electronic device performs model inference by using an initial model, and in units of network layers, counts intermediate data output by each network layer in the initial model. The electronic device calculates a truncated value of an activation value by using the intermediate data output by each network layer to obtain the truncated value of the activation value output by each network layer.

[0056] In some embodiments, the initial model can be a floating-point network model, which is specifically selected according to actual conditions, and the embodiments of the present disclosure are not limited.

[0057] In S102, the initial model is simulated and quantized based on the quantization mode of the target deployment platform to obtain a quantization data set. The simulated quantization is used to quantize and restore the activation values output by each network layer in the initial model, and then the activation values are input into the next network layer for inference. The quantization data set contains the truncated values of the activation values after simulated quantization of each network layer.

[0058] In the embodiments of the present disclosure, the electronic device can simulate and quantize the initial model based on the quantization mode of the target deployment platform to simulate the performance of the initial model after quantization using the quantization mode of the target deployment platform, and obtain a quantization data set.

[0059] In the embodiments of the present disclosure, during the simulated quantization process, the electronic device quantizes the activation values output by each network layer in the initial model, and then restores or dequantizes the quantized activation values. The restored or dequantized activation values are used as the input of the next network layer, so as to simulate the numerical precision of each network layer after quantization.

[0060] In the embodiments of the present disclosure, the electronic device calculates the truncated values of the activation values based on the simulated quantization of each network layer, that is, the quantized and restored activation values, to obtain the truncated values of the activation values after simulated quantization of each network layer as quantization data, and takes the set of quantization data of at least one network layer in the electronic device as a quantization data set.

[0061] In some embodiments, the electronic device can quantize the floating-point activation values to 8-bit integers, and then dequantize them to restore them to floating-point values, thereby completing the simulated quantization process.

[0062] In S103, a quantization error set is calculated based on the initial data set and the quantization data set, and the initial model is adjusted based on the quantization error set to obtain a quantization model.

[0063] In the embodiments of the present disclosure, the electronic device compares the quantization data set obtained by simulated quantization with the initial data set obtained by inference of the initial model with original precision to obtain a quantization error set. The electronic device can perform error analysis such as error distribution and error size based on the quantization error set, thereby locating the network layer with greater precision loss after quantization in the initial model or the distribution characteristics of errors in each network layer, and selecting a corresponding error adjustment mode based on the result of error analysis to adjust the network parameters of the initial model to obtain a quantization model.

[0064] It is understandable that quantization models are obtained through error adjustment. Therefore, compared with directly quantized models or mixed-precision models in related technologies, quantization models can achieve higher accuracy.

[0065] S104. Based on the truncation value of the activation value after simulation quantization for each network layer, generate quantization parameters; the quantization parameters are used to deploy the quantization model to the target deployment platform.

[0066] In this embodiment of the disclosure, the electronic device generates quantization parameters based on the truncation value of the activation value after simulation quantization of each network layer, and sends the quantization parameters and quantization model to the target deployment platform, so that the target deployment platform can deploy and apply the quantization model according to the quantization parameters.

[0067] In some embodiments, the electronic device can perform at least one of the following processes on the truncated activation values ​​of each network layer after simulated quantization, namely parameter filtering, serialization, and packaging, according to the preset quantization index of the target deployment platform, to generate quantization parameters.

[0068] For example, an electronic device can filter the truncated activation values ​​of each network layer based on the target deployment platform's different processing of truncation values, data format requirements, and network layer requirements. The filtered truncation activation values ​​are then serialized or packaged to generate quantization parameters in file format. In this way, when deploying the quantization model, the target deployment platform can implement a specific deployment process based on the quantization parameters to complete the application of the quantization model in a real-world scenario.

[0069] It is understood that in this embodiment of the disclosure, a quantized data set is obtained through simulated quantization. The initial data set is then compared with the quantized data set to obtain a quantization error set. This allows for error analysis based on the quantization error set and adjustment of the initial model, effectively improving the accuracy of the quantization model. Furthermore, based on the quantization mode of the target deployment platform, simulated quantization is performed on the initial model to generate quantization parameters for deployment on the target platform. This supports multi-platform deployment, allowing for model quantization using different quantization modes based on the different hardware and software configurations of the target deployment platform, generating different quantization parameters. Consequently, the quantized model can be deployed on the target deployment platform using the platform-specific quantization parameters, achieving high-precision, multi-platform model quantization and improving the versatility of the quantization model application.

[0070] In some embodiments, based on Figure 1 ,like Figure 2 As shown, Figure 1 S101 can be achieved by executing the process S1011-S1013, which will be explained in conjunction with each step.

[0071] S1011, acquire quantization calibration data.

[0072] In the embodiments of the present disclosure, the quantization calibration data is a preset sample data set, which is used to verify or calibrate the inference result of model quantization. For example, the quantization calibration data can be a test set or a verification set containing a large number of picture samples, and the specific selection is based on actual conditions, which is not limited in the embodiments of the present disclosure.

[0073] S1012, performing model inference on the quantization calibration data through each network layer in the initial model to obtain at least one activation value output by each network layer.

[0074] In the embodiments of the present disclosure, the electronic device acquires quantization calibration data, performs forward propagation model inference and prediction on the quantization calibration data through each network layer in the initial model, and counts the intermediate data output by each network layer as at least one activation value.

[0075] In some embodiments, the electronic device can perform model inference on the quantization calibration data based on any one of quantization algorithms such as maximum and minimum value, KL (Kullback Leibler) divergence quantization, and EQ (Easy Quant) quantization. The specific selection is based on actual conditions, which is not limited in the embodiments of the present disclosure.

[0076] S1013, for at least one activation value of each network layer, performing statistical truncation processing by using at least one preset statistical algorithm to obtain a truncated value of the activation value corresponding to each network layer as an initial data set.

[0077] In the embodiments of the present disclosure, at least one preset statistical algorithm is pre-deployed on the electronic device. For each network layer, the electronic device can select any one statistical algorithm from the at least one preset statistical algorithm to perform statistical truncation processing on at least one activation value of each network layer, so that the calculation of the truncated value of the activation value by using different statistical algorithms for different network layers can be realized, and the truncated value of the activation value corresponding to each network layer is obtained as an initial data set.

[0078] In some embodiments, S1013 can include any one of the following:

[0079] For each network layer, the maximum value is counted from at least one activation value as the truncated value of the activation value, or the minimum value is counted from at least one activation value as the truncated value of the activation value, or the median of at least one activation value is calculated, and the variance is calculated based on the median, such as calculating the number of times of variance to obtain the truncated value of the activation value.

[0080] It can be understood that, by using any one of the preset statistical algorithms, the calculation of the truncation value of the activation value of different network layers can select a more suitable statistical algorithm according to different activation values output by different network layers, obtain a higher-precision activation value truncation value, and greatly improve the precision of model quantization compared with the statistical algorithm supporting only the maximum value / minimum value in the prior art.

[0081] In some embodiments, based on Figure 1 or Figure 2 As Figure 3 As shown in S102, S102 can be implemented by S1021-S1023, which will be described in combination with each step.

[0082] S1021, for each network layer in the initial model, obtaining a quantization scale corresponding to each network layer according to the truncation value of the activation value corresponding to each network layer.

[0083] In the embodiments of the present disclosure, in the case of analog quantization, the electronic device obtains the quantization scale corresponding to each network layer based on the truncation value of the activation value corresponding to each network layer obtained by the statistical algorithm, that is, the calculation of the scaling factor used in the quantization process, to obtain the quantization scale corresponding to each network layer.

[0084] In some embodiments, taking the quantization target as a signed 8-bit integer type as an example, the electronic device can divide the truncation value of the activation value corresponding to each network layer by 128 to obtain the quantization scale corresponding to each network layer.

[0085] S1022, using the i-th layer network in the initial model to infer the i-1-th quantized data output by the i-1-th layer to obtain the i-th initial activation value; wherein i is a positive integer greater than or equal to 2; the first quantized data is obtained by inferring the quantization calibration data by the first layer network in the initial model, and the first initial activation value obtained by the inference is quantized and scaled.

[0086] S1023, according to the quantization scale corresponding to the i-th layer network, combining the quantization mode and the preset quantization precision, quantizing and scaling the i-th initial activation value to obtain the i-th quantized data, realizing analog quantization of the i-th layer network, until analog quantization of each network layer in the initial model is realized, to obtain a set of quantized data.

[0087] In the embodiments of the present disclosure, for the first network layer in the initial model, the electronic device takes the quantization calibration data as the input data of the first network layer, and uses the first network layer to quantize and infer the quantization calibration data to obtain the initial activation value output by the first network layer.

[0088] In the embodiments of the present disclosure, the electronic device performs quantization processing and scale restoration on the activation value output by the first network layer according to the quantization scale corresponding to the first network layer, in combination with the quantization mode and the preset quantization precision, to obtain first quantization data. The first quantization data is taken as input data of the second network layer, and the subsequent analog quantization process is continued.

[0089] In the embodiments of the present disclosure, for each network layer after the first network layer, the quantization data after analog quantization of the previous network layer is taken as input data of the network layer, to obtain initial activation value corresponding to the network layer. Then, the initial activation value corresponding to the network layer is subjected to quantization processing and scale restoration according to the quantization scale corresponding to the network layer, in combination with the quantization mode and the preset quantization precision, to obtain quantization data corresponding to the network layer. The electronic device performs the above analog quantization process until analog quantization of each network layer in the initial model is completed, to obtain quantization data corresponding to each network layer as a quantization data set.

[0090] In some embodiments, the quantization mode includes a zero point coefficient based on Figure 3 S1023 can be implemented by S301-S304, which will be described in combination with each step.

[0091] S301, taking the integer of the ratio of the i th initial activation value to the quantization scale to obtain the i th initial quantization value.

[0092] S302, adjusting the i th initial quantization value by using the zero point coefficient to obtain the i th intermediate quantization value.

[0093] S303, truncating the i th intermediate quantization value by using the preset quantization precision to obtain the i th quantization value, and completing the quantization processing process.

[0094] S304, taking the product of the i th quantization value and the quantization scale as the i th quantization data, and completing the scale restoration process.

[0095] In the embodiments of the present disclosure, the electronic device can calculate the quantization data corresponding to each network layer by formula (1) as follows:

[0096]

[0097] In formula (1), D qFP32 is the input data corresponding to each network layer itself, for the first network layer, FP32 can be the quantization calibration data, and for each network layer after the first network layer, FP32 can be the quantization data corresponding to the previous network layer; scale is the quantization scale calculated in S1021; Round represents the rounding operation, such as rounding; zero_ponit is the zero point coefficient, which can be different for different target deployment platforms of software and hardware configurations; min and max represent the number of quantization bits, that is, the quantization precision, such as min and max of 8-bit quantization being -128, 127; Clip represents truncation with the maximum and minimum values. The electronic device performs analog quantization on each network layer through formula (1) to obtain the quantization data corresponding to each network layer as the quantization data set.

[0098] It can be understood that, by performing the analog quantization process based on the zero point coefficient of the target deployment platform, the electronic device can obtain the quantization effect of the initial model on the target deployment platform without actually quantizing the initial model, so that different analog quantization processes can be performed for different target deployment platforms to obtain multiple quantization data sets corresponding to multiple platforms, and model quantization simulation for multiple platforms can be realized through one set of model quantization tools.

[0099] In some embodiments, based on Figure 1 or Figure 3 As shown in FIG. 10, S103 can be implemented by performing S1031-S1033, which will be described in combination with each step. Figure 4

[0100] S1031, calculate the cosine distance between each initial data in the initial data set and each quantization data in the quantization data set as the quantization error corresponding to each network layer, thereby obtaining the quantization error set.

[0101] In the embodiments of the present disclosure, the electronic device can calculate the cosine distance between each initial data in the initial data set and each quantization data in the quantization data set, and take the cosine distance as the error between each initial data and each quantization data, thereby obtaining the quantization error set.

[0102] In some embodiments, the electronic device can also calculate and evaluate the error between the initial data and the quantization data by other error calculation methods, which are selected according to actual conditions, and the embodiments of the present disclosure are not limited.

[0103] S1032, in the case where it is determined that the preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the quantization error set, determine the error adjustment mode according to the platform type of the target deployment platform. ​

[0104] In the embodiments of the present disclosure, when the set of quantization errors is obtained, the electronic device can perform at least one of error value analysis and error distribution analysis based on the set of quantization errors, to evaluate the quantization precision of the initial model by the set of quantization errors, and locate the network layer position that most affects the quantization precision of the model. For example, when the number of quantization errors with large values in the set of quantization errors is too large, and / or the set of quantization errors contains quantization errors with excessively large values, it indicates that the precision of the simulation quantization cannot meet the needs of actual deployment and application. The electronic device can determine that the set of quantization errors meets the preset adjustment condition, and then determine an error adjustment manner according to the platform type of the target deployment platform, to adjust and improve the precision of the initial model.

[0105] In some embodiments, the electronic device can determine the maximum value of the quantization errors in the set of quantization errors, that is, the maximum quantization error contained in each network layer. When the maximum value of the quantization errors is greater than the first preset error threshold, it indicates that the maximum error of the initial model is too large to meet the quantization precision requirement, and the electronic device determines that the preset adjustment condition is met.

[0106] In some embodiments, the electronic device can count the number of quantization errors greater than the second preset error threshold in the set of quantization errors; here, the second preset error threshold is less than the first preset error threshold. When the number of quantization errors greater than the second preset error threshold is greater than the error number threshold, it indicates that the quantization precision of most network layers in the initial model is not ideal, and the electronic device determines that the preset adjustment condition is met.

[0107] In some embodiments, the error adjustment manner includes at least one of a model cross-layer averaging algorithm, a bias correction algorithm, and an adaptive rounding algorithm.

[0108] Here, the model cross-layer averaging algorithm is used to average the value distribution range of each channel when the deployment backend is in a non-per-channel quantization mode, to achieve a precision close to per-channel quantization, and to solve the precision problem caused by too large difference in the value distribution range of each channel on non-per-channel quantization hardware. The bias correction algorithm is a lightweight and widely applicable algorithm, which can be used to reduce the quantization precision of the model and has a fast speed. The adaptive rounding algorithm can best fit the errors of the quantization model and the full-precision model.

[0109] In some embodiments, the electronic device can evaluate the quantization precision of the initial model by any one of the above-mentioned maximum value of the quantization errors and the number of quantization errors, to determine whether the preset adjustment condition is met, or can evaluate and judge by the combination of the two manners, which is selected according to actual conditions, and the embodiments of the present disclosure are not limited.

[0110] In some embodiments, the electronic device can, in a case where it is determined that the preset adjustment condition is met, obtain a type of quantization mode supported by the platform according to the platform type of the target deployment platform, and then select and determine a suitable error adjustment algorithm from at least one error adjustment algorithm pre-integrated on the electronic device as the error adjustment manner according to the type of quantization mode supported by the platform.

[0111] S1033, adjusting the network parameters of the initial model by using the error adjustment manner to obtain a quantization model.

[0112] In the embodiments of the present disclosure, the electronic device can adjust the network parameters such as network weights of the initial model by using the error adjustment manner, obtain new model parameters and save them, and obtain a quantization model, thereby significantly reducing the precision error of the quantization model.

[0113] It can be understood that the embodiments of the present disclosure can determine a suitable error adjustment manner by providing a selection of multiple error adjustment manners to reduce the quantization error of the model. In actual use, one or more algorithms can be selected for use, such as a model with a large precision loss that can use a higher level algorithm, and a model with a smaller precision loss that can use a lower level algorithm. Reasonable use of error adjustment algorithms can significantly improve the precision of the quantized model.

[0114] In some embodiments, in a case where it is determined that the preset adjustment condition is met by performing error value analysis and / or error distribution analysis on the quantization error set, the electronic device can determine a target network layer with a more concentrated quantization error distribution and / or a larger impact on model precision based on the analysis results of the error value analysis and / or error distribution analysis, and then select an error adjustment algorithm for the target network layer to adjust the network parameters of the initial model to obtain a quantization model. In this way, by locating the target network layer through error analysis and then adjusting it, the computational load of adjusting the entire initialized network can be reduced, and the efficiency of model quantization and adjustment can be improved.

[0115] In some embodiments, after S1031, the electronic device can further perform S1034 to adjust the network parameters of the initial model to obtain a quantization model, as shown in Figure 5

[0116] S1034, in a case where it is determined that the preset adjustment condition is met by performing error value analysis and / or error distribution analysis on the quantization error set, adjusting the network parameters of the initial model by comparing tensors of the initial data set and the quantization data set to obtain a quantization model.

[0117] ​In this embodiment of the disclosure, the electronic device determines that the quantization error set has reached the preset adjustment condition, which is consistent with the description in S1032, and will not be repeated here. When the quantization error set reaches the preset adjustment condition, the electronic device can also adjust the network parameters of the initial model by comparing and analyzing the initial data set and the quantized data set from the tensor dimension, thereby obtaining the quantized model.

[0118] In this embodiment of the disclosure, tensors represent the data processed by the network model. Tensors have the attributes of numerical value, dimension, and size. The electronic device can compare the differences between the tensors output by the initial model and the tensors output by the quantization model based on the dimensions of the tensors, locate the network location and cause that affects the quantization accuracy of the model, and select the corresponding error adjustment algorithm to adjust the network parameters of the initial model based on the cause location.

[0119] In some embodiments, based on Figure 5 ,like Figure 6 As shown, S1034 can be implemented by executing S401-S403, and will be explained in conjunction with each step.

[0120] S401. Tensor distribution information is obtained by comparing the tensor numerical distributions of the initial data set and the quantized data set; and / or, quantization grouping information is obtained by comparing the tensor scale information of the initial data set and the quantized data set; the tensor scale characterizes the grouping form used for quantization.

[0121] In this embodiment of the disclosure, the electronic device can obtain the numerical distribution of the tensor corresponding to the initial data set and the numerical distribution of the tensor corresponding to the quantized data set. By comparing the numerical distribution of the tensors, the distribution characteristics of the tensors output by the initial model and the quantized model can be obtained and compared to obtain tensor distribution information. Then, based on the tensor distribution information, the network layer with a large difference in the numerical distribution of the tensors output by the initial model and the quantized model can be located.

[0122] In this embodiment of the disclosure, the electronic device can also compare the tensor scale information of the initial data set with the tensor scale information of the quantized data set to obtain the grouping form of each network layer during the quantization process, which serves as the quantization grouping information, thereby determining whether the quantization error is caused by the quantization grouping and thus locating the cause of the error.

[0123] S402. Use tensor distribution information and / or quantization grouping information as error performance information, and determine the error adjustment method based on the error performance information, or based on a combination of error performance information and the platform type of the target deployment platform.

[0124] In the embodiments of the present disclosure, the electronic device takes the tensor distribution information and / or the quantization grouping information as error manifestation information, and then can determine the error adjustment manner based on the error manifestation information itself, or based on the combination of the error manifestation information and the platform type of the target deployment platform.

[0125] In some embodiments, determining the error adjustment manner based on the error manifestation information or based on the combination of the error manifestation information and the platform type of the target deployment platform in S402 can include: determining an error adjustment algorithm acting on the quantization error distribution dimension and / or the quantization grouping dimension as the error adjustment manner based on the numerical distribution information of the quantization error and / or the quantization grouping information contained in the error manifestation information; or determining the error adjustment algorithm as the error adjustment manner based on the error manifestation information and the network level scale represented by the platform type.

[0126] In some embodiments, determining the error adjustment manner based on the error manifestation information or based on the combination of the error manifestation information and the platform type of the target deployment platform in S402 can be implemented by at least one of S402-11 to S402-12, S402-21, and S402-31 to S402-33, which will be described in combination with each step.

[0127] S402-11, obtaining the distribution feature difference of the tensor numerical values of each channel of the initial data set and the quantized data set according to the tensor distribution information in the error manifestation information.

[0128] S402-12, determining the model cross-layer averaging algorithm as the error adjustment manner in the case that the distribution feature difference meets the preset tensor distribution adjustment condition.

[0129] In the embodiments of the present disclosure, the preset tensor distribution adjustment condition can be that the difference in the numerical value distribution dimension of the tensors output by the initial data set and the quantized data set is greater than or equal to a preset tensor distribution difference.

[0130] In some embodiments, each network layer in the initial model and the quantized model corresponds to at least one channel. The electronic device can obtain the numerical value distribution range of the tensors output by each channel in each network layer of the initial model and the numerical value distribution range of the tensors output by each channel in each network layer of the quantized model according to the tensor distribution information, and obtain the distribution feature difference of the tensors by comparing the numerical value distribution range of the tensors on each channel. In the case that the distribution feature difference meets the preset tensor distribution adjustment condition, the model cross-layer averaging algorithm is determined as the error adjustment manner, so as to utilize the model cross-layer averaging algorithm to average the numerical value distribution range of the tensors output by each channel of the initial model, and reduce the precision problem caused by too large difference in the numerical value distribution range of each channel.

[0131] S402-21, in the case that the quantization grouping information in the error performance information indicates that there are at least two quantization groupings, determining a bias correction algorithm as the error adjustment manner.

[0132] In the embodiments of the present disclosure, in the case that the quantization grouping information in the error performance information indicates that there are at least two quantization groupings, the electronic device can determine that the error is caused by the grouping quantization, and the grouping granularity of the weight in the quantization process can affect the quantization precision. The electronic device determines the bias correction algorithm as the error adjustment manner to reduce the influence of the grouping granularity on the quantization precision through the bias correction algorithm, and improve the quantization precision.

[0133] S402-31, in the case that the distribution feature difference satisfies the preset tensor distribution adjustment condition, or the quantization grouping information indicates that there are at least two quantization groupings, determining a preset quantization precision corresponding to the platform type.

[0134] In the embodiments of the present disclosure, in the case that the distribution feature difference satisfies the preset tensor distribution adjustment condition, or the quantization grouping information indicates that there are at least two quantization groupings, the electronic device can also determine the requirement of the target deployment platform for the quantization precision according to the platform type of the target deployment platform, that is, the preset quantization precision.

[0135] S402-32, in the case that the preset quantization precision is higher than the preset precision threshold, determining an adaptive rounding algorithm as the error adjustment manner.

[0136] In the embodiments of the present disclosure, in the case that the preset quantization precision is higher than the preset precision threshold, it indicates that the target deployment platform has higher requirement for the precision. The electronic device can determine the adaptive rounding algorithm as the error adjustment manner to use the training adjustment process contained in the adaptive rounding algorithm to maximize the fitting of the quantization error of the quantization model and the initial model, so as to meet the requirement of the high-precision target deployment platform.

[0137] S403, adjusting the network parameters of the initial model by using the error adjustment manner to obtain a quantization model.

[0138] Here, the process of S403 is consistent with that described in S1033, which will not be repeated here.

[0139] It can be understood that through the support of multiple network layer scales and multiple algorithms, different degrees of model quantization precision loss can be coped with, and algorithms can be selected according to the characteristics of different platforms to improve the model quantization precision.

[0140] Next, the embodiments of the present disclosure will be described in combination with Figure 7The application discloses the application of the model quantization method in actual scenes. The model quantization method can be applied to a quantization tool to quantize a trained original full-precision model through the quantization tool and generate quantization parameters required for deployment to a target deployment platform.

[0141] S501, an activation value algorithm is selected to perform activation value statistics and calculation of a calculated value of the activation value.

[0142] In S501, the electronic device obtains an initial model that needs to be quantized and quantization calibration data that have been trained, and then performs model inference several times according to different activation value calibration algorithms (Minmax / Hist / KL) and the like, saves intermediate data output by each network layer as statistical quantities required by the algorithms, obtains total statistical results of quantization data at each layer of the model after the inference ends, and then generates a clipping value of the activation value corresponding to each network layer according to the activation value calibration algorithm.

[0143] S502, simulated quantization is performed according to a quantization mode of a backend to be deployed, error analysis under the model quantization mode is generated, and precision analysis results are obtained.

[0144] In S502, the backend to be deployed is a target deployment platform, and the electronic device performs simulated quantization on the initial model in combination with quantization mode information such as a zero coefficient determined by different platform hardware and chips themselves, under the condition that the clipping value of the activation value of each network layer is obtained. Exemplarily, the simulated quantization adopts a linear quantization mode, and quantization data is calculated through formula (1).

[0145] In S502, the simulated quantization process can give a comparison between network output after quantization deployment of the network model and network output during training, to measure the precision loss of the model quantization. Meanwhile, the tool can give a quantized output tensor obtained through layer-by-layer forward calculation of the model, and compare the output tensor with an output tensor of an original full-precision model, that is, the initial model, in terms of cosine error analysis, numerical distribution of the tensor, and dimensional information of the tensor, to quickly find a position and a reason that most affect the precision of the model quantization, as the precision analysis results.

[0146] S503, an error adjustment algorithm for adjusting model weights is adopted according to the precision analysis results.

[0147] In S503, the electronic device adjusts the weights of the initial model by selecting a suitable algorithm from a plurality of error adjustment algorithms integrated in advance according to the precision analysis results, under the condition that the precision analysis results represent that a preset error adjustment condition is reached, to obtain a quantization model.

[0148] S504, quantization parameters corresponding to the target deployment platform are generated according to a platform type of the target deployment platform.

[0149] In S504, the electronic device performs parameter screening, packaging, serialization, etc. on the model parameters of the quantization model according to the platform type of the target deployment platform, to generate quantization parameters corresponding to the target deployment platform.

[0150] It can be understood that the quantization tool provided by the embodiments of the present disclosure can support multi-platform deployment, that is, the same model and data can be flexibly deployed on multiple platforms without additional cost. Moreover, the embodiments of the present disclosure support multiple quantization calibration algorithms, which can be flexibly provided to users for experiments and attempts, greatly improving the accuracy of the model and the robustness and convenience of the tool. Moreover, the embodiments of the present disclosure support error analysis and numerical analysis functions of model quantization accuracy, so that users can accurately locate the network layer with larger error when encountering model accuracy problems, to make targeted further accuracy adjustment to improve accuracy, thereby realizing a high-precision, multi-platform, and automatic offline quantization model production tool.

[0151] The present disclosure also provides a model quantization device, Figure 8 A structural schematic diagram of the model quantization device provided by the embodiments of the present disclosure is shown in Figure 8 As shown in the figure, the model quantization device 1 comprises:

[0152] An inference module 11 is configured to perform model inference using an initial model to be quantized to obtain an initial data set; the initial data set contains the truncated values of the activation values output by each network layer in the initial model;

[0153] An analog quantization module 12 is configured to perform analog quantization on the initial model based on the quantization mode of a target deployment platform to obtain a quantization data set; the analog quantization is used to quantize and restore the activation values output by each network layer in the initial model, and then the quantized and restored activation values are input into the next network layer for inference; the quantization data set contains the truncated values of the analog-quantized activation values of each network layer;

[0154] An error adjustment module 13 is configured to calculate a quantization error set based on the initial data set and the quantization data set, and adjust the initial model based on the quantization error set to obtain a quantization model;

[0155] A parameter generation module 14 is configured to generate quantization parameters based on the truncated values of the analog-quantized activation values of each network layer; the quantization parameters are used to deploy the quantization model to the target deployment platform.

[0156] In some embodiments, the inference module 11 is further configured to obtain quantization calibration data; perform model inference on the quantization calibration data through each network layer in the initial model to obtain at least one activation value output by each network layer; and for the at least one activation value of each network layer, perform statistical truncation processing on the at least one activation value using at least one preset statistical algorithm to obtain a truncated value of the activation value corresponding to each network layer as the initial data set.

[0157] In some embodiments, the truncated value of the activation value corresponding to each network layer includes any one of the following:

[0158] a maximum value in the at least one activation value, a minimum value in the at least one activation value, or a variance value of a median of the at least one activation value.

[0159] In some embodiments, the simulation quantization module 12 is further configured to, for each network layer in the initial model, obtain a quantization scale corresponding to each network layer according to the truncated value of the activation value corresponding to each network layer; perform inference on an (i-1)th quantized data output by an (i-1)th layer using an i-th layer in the initial model to obtain an i-th initial activation value; wherein i is a positive integer greater than or equal to 2; the first quantized data is obtained by performing inference on the quantization calibration data using a first layer in the initial model, and performing quantization processing and scale restoration on a first initial activation value obtained by inference; performing quantization processing and scale restoration on the i-th initial activation value according to the quantization scale corresponding to the i-th layer, in combination with the quantization mode and a preset quantization precision, to obtain an i-th quantized data, thereby realizing simulation quantization on the i-th network layer, until simulation quantization is realized on each network layer in the initial model, thereby obtaining the quantized data set.

[0160] In some embodiments, the quantization mode includes a zero point coefficient, and the simulation quantization module 12 is further configured to: take an integer of a ratio of the i-th initial activation value to the quantization scale to obtain an i-th initial quantized value; adjust the i-th initial quantized value using the zero point coefficient to obtain an i-th intermediate quantized value; perform truncation on the i-th intermediate quantized value using the preset quantization precision to obtain an i-th quantized value, thereby completing the quantization processing process; and multiply the i-th quantized value by the quantization scale to obtain the i-th quantized data, thereby completing the scale restoration process.

[0161] In some embodiments, the error adjustment module 13 is configured to calculate a cosine distance between each initial data in the initial data set and each quantized data in the quantized data set as a quantization error corresponding to each network layer, thereby obtaining the quantization error set.

[0162] In some embodiments, the error adjustment module 13 is further configured to, in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the set of quantization errors, determine an error adjustment manner according to a platform type of the target deployment platform; and adjust the network parameters of the initial model by using the error adjustment manner to obtain the quantization model.

[0163] In some embodiments, the error adjustment module 13 is further configured to, in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the set of quantization errors, adjust the network parameters of the initial model based on an analysis result of the error numerical analysis and / or the error distribution analysis to obtain the quantization model.

[0164] In some embodiments, the error adjustment module 13 is further configured to, in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the set of quantization errors, adjust the network parameters of the initial model by performing tensor comparison on the set of initial data and the set of quantization data to obtain the quantization model.

[0165] In some embodiments, the error adjustment module 13 is further configured to obtain tensor distribution information by comparing tensor numerical distribution of the set of initial data and the set of quantization data; and / or obtain quantization grouping information by comparing tensor scale information of the set of initial data and the set of quantization data; the tensor scale represents a grouping form used for quantization; take the tensor distribution information and / or the quantization grouping information as error performance information, and determine an error adjustment manner based on the error performance information, or based on a combination of the error performance information and a platform type of a target deployment platform; and adjust the network parameters of the initial model by using the error adjustment manner to obtain the quantization model.

[0166] In some embodiments, the error adjustment module 13 is further configured to obtain a distribution feature difference of tensor values of each channel between the initial data set and the quantized data set according to tensor distribution information in the error performance information; determine a model cross-layer averaging algorithm as the error adjustment manner in a case where the distribution feature difference meets a preset tensor distribution adjustment condition; and / or determine a bias correction algorithm as the error adjustment manner in a case where quantization grouping information in the error performance information indicates that there are at least two quantization groups; and / or determine a preset quantization precision corresponding to the platform type in a case where the distribution feature difference meets the preset tensor distribution adjustment condition or the quantization grouping information indicates that there are at least two quantization groups; and determine an adaptive rounding algorithm as the error adjustment manner in a case where the preset quantization precision is higher than a preset precision threshold.

[0167] In some embodiments, the error adjustment module 13 is further configured to determine that a preset adjustment condition is met in a case where a maximum value of quantization errors included in the quantization error set is greater than a first preset error threshold; and / or determine that the preset adjustment condition is met in a case where a number of quantization errors greater than a second preset error threshold in the quantization error set is greater than an error number threshold; the second preset error threshold is less than the first preset error threshold.

[0168] In some embodiments, the parameter generation module 14 is configured to perform at least one of parameter screening, serialization, and packaging on the truncated value of the simulated quantized activation value of each network layer according to a preset quantization index of a target deployment platform, to generate the quantization parameter.

[0169] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0170] The present disclosure also provides an electronic device, Figure 9 The structure schematic diagram of the electronic device provided by the embodiments of the present disclosure is shown in Figure 9 As shown in the figure, the electronic device 2 includes a memory 22 and a processor 23, wherein the memory 22 and the processor 23 are connected through a communication bus 24; the memory 22 is configured to store executable instructions; the processor 23 is configured to execute the executable instructions stored in the memory 22 to implement the method provided by the embodiments of the present disclosure, for example, the model quantization method provided by the embodiments of the present disclosure.

[0171] The embodiment of the present disclosure provides a computer readable storage medium, which stores executable model quantization instructions, and when the processor 23 executes the executable model quantization instructions, the method provided by the embodiment of the present disclosure is implemented, for example, the model quantization method provided by the embodiment of the present disclosure.

[0172] In some embodiments of the present disclosure, the storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM, etc. memory; it can also be various devices including one or any combination of the above-mentioned memories.

[0173] In some embodiments of the present disclosure, the executable model quantization instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0174] As an example, the executable model quantization instructions can but not necessarily correspond to a file in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0175] As an example, the executable model quantization instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed in multiple sites and interconnected through a communication network.

[0176] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as methods, systems or computer program products. Therefore, the present disclosure can take the form of hardware embodiments, software embodiments or embodiments combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program code.

[0177] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0178] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0179] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0180] The above description is only preferred embodiments of the present disclosure, not intended to limit the protection range of the present disclosure.

Claims

1. A model quantization method, characterized in that, The method comprises the following steps: model inference is performed on the initial model to be quantized to obtain an initial data set; the initial data set comprises a truncated value of an activation value output by each network layer in the initial model; simulated quantization is performed on the initial model based on a quantization mode of a target deployment platform to obtain a quantization data set; the simulated quantization is used to quantize and restore the activation value output by each network layer in the initial model, and the quantization data set comprises a truncated value of the activation value of each network layer after simulated quantization; the quantization mode comprises a zero point coefficient, and the zero point coefficient is determined based on hardware and a chip of the target deployment platform; a quantization error set is calculated based on the initial data set and the quantization data set, and the initial model is adjusted based on the quantization error set to obtain a quantization model; a quantization parameter is generated based on the truncated value of the activation value of each network layer after simulated quantization, and the quantization parameter is used to deploy the quantization model to the target deployment platform.

2. The method of claim 1, wherein, The method of performing model inference on the initial model to be quantized to obtain an initial data set comprises the following steps: obtaining quantization calibration data; model inference is performed on the initial model to be quantized to obtain an initial data set; for at least one activation value of each network layer, statistical truncation processing is performed on the at least one activation value by using at least one preset statistical algorithm to obtain a truncated value of the activation value corresponding to each network layer as the initial data set.

3. The method of claim 2, wherein, The truncated value of the activation value corresponding to each network layer comprises any one of the following: a maximum value in the at least one activation value, a minimum value in the at least one activation value, or a variance value of a median of the at least one activation value.

4. The method according to any one of claims 1 to 3, characterized in that, The method of performing simulated quantization on the initial model based on a quantization mode of a target deployment platform to obtain a quantization data set comprises the following steps: for each network layer in the initial model, a quantization scale corresponding to the each network layer is obtained according to a truncated value of an activation value corresponding to the each network layer; the i-th initial activation value is obtained by using an i-th network layer in the initial model to perform inference on an i-1-th quantization data output by an i-2-th network layer, where i is a positive integer greater than or equal to 2, and the first quantization data is obtained by performing inference on the quantization calibration data by using a first network layer in the initial model and performing quantization processing and scale restoration on a first initial activation value obtained by inference; the i-th quantization data is obtained by performing quantization processing and scale restoration on the i-th initial activation value according to the quantization scale corresponding to the i-th network layer, in combination with the quantization mode and a preset quantization precision, to realize simulated quantization on the i-th network layer, and the simulated quantization is performed on each network layer in the initial model to obtain the quantization data set.

5. The method of claim 4, wherein, The quantization mode comprises a zero point coefficient, and the method of obtaining the i-th quantization data by performing quantization processing and scale restoration on the i-th initial activation value according to the quantization scale corresponding to the i-th network layer, in combination with the quantization mode and a preset quantization precision comprises the following steps: integer dividing the ratio of the i th initial activation value and the quantization scale to obtain an i th initial quantization value; adjusting the i th initial quantization value by using a zero point coefficient to obtain an i th intermediate quantization value; truncating the i th intermediate quantization value by using the preset quantization precision to obtain an i th quantization value, and completing a quantization process; multiplying the i th quantization value and the quantization scale to obtain the i th quantization data, and completing a scale restoration process.

6. The method according to claim 1 or 5, characterized in that, The calculating a quantization error set based on the initial data set and the quantization data set comprises: calculating a cosine distance between each initial data in the initial data set and each quantization data in the quantization data set as a quantization error corresponding to each network layer, thereby obtaining the quantization error set.

7. The method of claim 1, wherein, The adjusting the initial model based on the quantization error set to obtain a quantization model comprises: determining an error adjustment mode according to a platform type of the target deployment platform in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the quantization error set; adjusting network parameters of the initial model by using the error adjustment mode to obtain the quantization model.

8. The method of claim 1, wherein, The adjusting the initial model based on the quantization error set to obtain a quantization model comprises: adjusting network parameters of the initial model based on an analysis result of error numerical analysis and / or error distribution in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the quantization error set, thereby obtaining the quantization model.

9. The method of claim 1, wherein, The adjusting the initial model based on the quantization error set to obtain a quantization model comprises: adjusting network parameters of the initial model by performing tensor comparison on the initial data set and the quantization data set in a case where it is determined that a preset adjustment condition is reached by performing error numerical analysis and / or error distribution analysis on the quantization error set, thereby obtaining the quantization model.

10. The method of claim 9, wherein, The adjusting the initial model based on the quantization error set to obtain a quantization model comprises: obtaining tensor distribution information by comparing tensor numerical distribution of the initial data set and the quantization data set; and / or, obtaining quantization grouping information by comparing tensor scale information of the initial data set and the quantization data set; the tensor scale represents a grouping form for quantization; taking the tensor distribution information and / or the quantization grouping information as error performance information, and determining an error adjustment mode based on the error performance information or based on a combination of the error performance information and a platform type of the target deployment platform; adjusting network parameters of the initial model by using the error adjustment mode to obtain the quantization model.

11. The method of claim 10, wherein, The determining an error adjustment mode based on the error performance information or based on a combination of the error performance information and a platform type of the target deployment platform comprises at least one of the following: obtaining a distribution characteristic difference of tensor values of each channel between the initial data set and the quantized data set according to tensor distribution information in the error performance information; in a case where the distribution characteristic difference meets a preset tensor distribution adjustment condition, determining a model cross-layer averaging algorithm as the error adjustment manner; in a case where quantization grouping information in the error performance information indicates that there are at least two quantization groups, determining a bias correction algorithm as the error adjustment manner; in a case where the distribution characteristic difference meets a preset tensor distribution adjustment condition, or the quantization grouping information indicates that there are at least two quantization groups, determining a preset quantization precision corresponding to the platform type; in a case where the preset quantization precision is higher than a preset precision threshold, determining an adaptive rounding algorithm as the error adjustment manner.

12. The method according to any one of claims 7-11, characterized in that, determining that a preset adjustment condition is reached through error value analysis and / or error distribution analysis on the quantization error set, including at least one of the following: in a case where a maximum value of the quantization error included in the quantization error set is greater than a first preset error threshold, determining that the preset adjustment condition is reached; in a case where a number of quantization errors greater than a second preset error threshold in the quantization error set is greater than an error number threshold, determining that the preset adjustment condition is reached; the second preset error threshold is less than the first preset error threshold.

13. The method of any one of claims 1, 7 or 11, wherein the generating the quantization parameter based on the truncated value of the simulated quantized activation value of each network layer comprises: performing at least one of parameter screening, serialization, and packaging on the truncated value of the simulated quantized activation value of each network layer according to a preset quantization index of a target deployment platform to generate the quantization parameter.

14. A model quantization apparatus, comprising: comprising: an inference module configured to perform model inference on an initial model to be quantized to obtain an initial data set; the initial data set includes truncated values of activation values output by each network layer of the initial model; a simulated quantization module configured to perform simulated quantization on the initial model based on a quantization mode of a target deployment platform to obtain a quantized data set; the simulated quantization is used to quantize and restore the activation values output by each network layer of the initial model before entering a next network layer for inference, and the quantized data set includes truncated values of simulated quantized activation values of each network layer; the quantization mode includes a zero point coefficient, and the zero point coefficient is determined based on hardware and chips of the target deployment platform; an error adjustment module configured to calculate a quantization error set based on the initial data set and the quantized data set, and adjust the initial model based on the quantization error set to obtain a quantized model; a parameter generation module configured to generate a quantization parameter based on the truncated value of the simulated quantized activation value of each network layer, and the quantization parameter is used to deploy the quantized model to the target deployment platform.

15. An electronic device, comprising: comprising: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, A computer program or instructions are stored, which when executed by a processor implement the method of any of claims 1 to 13.

17. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by a processor to implement the method of any of claims 1 to 13.

Images