Model quantization method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a model quantization method, device, electronic device and storage medium, which relate to the field of deep learning technology. The method includes: obtaining the weight matrix to be quantized of the network layer to be quantized in the original model; dividing each weight in the weight matrix to be quantized into multiple weight groups; calculating the first model loss corresponding to each weight group; determining the current weight group to be processed in the order of the corresponding first model loss from large to small; taking the minimum second model loss corresponding to the current weight group to be processed as the optimization goal, determining the target quantization result of the current weight group to be processed based on the target quantization format; determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; obtaining the quantized target model based on the target quantization result of each weight group. In this way, the original model can be quantized, and the model loss of the quantized target model is small.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a model quantization method, device, electronic device and storage medium. Background Art

[0002] In the field of deep learning technology, trained deep learning models can be deployed on the platform, and then used to process business data on the platform. For example, an LLM (Large Language Model) trained based on a large amount of text data can process user input text and answer questions in the input text.

[0003] However, the model parameters in the trained deep learning model are in Float (floating point) format, for example, Float16 format. If the platform uses the deep learning model to process business data, a large number of floating point operations are required, which consumes a large amount of computing resources of the platform. If the computing resources of the platform are limited, the deep learning model will have problems such as slow computing speed and long response time. To solve the above problems, the model needs to be quantized. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model quantization method, device, electronic device and storage medium to obtain a quantized target model with a small model loss. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a model quantization method is first provided, the method comprising:

[0006] For each network layer to be quantized in the original model, obtain the weight matrix of the network layer to be quantized in floating point format as the weight matrix to be quantized; group the weights in the weight matrix to be quantized to obtain multiple weight groups; wherein the weights contained in a weight group correspond to the same input channel of the network layer to be quantized; for each weight group, use the target quantization coefficient of the weight group to calculate the model loss when the weight group is quantized to the target quantization format according to the activation-aware weight quantization algorithm, as the first model loss corresponding to the weight group; wherein, among the alternative quantization coefficients, the model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group is the smallest; the target quantization format is determined based on the quantization format supported by the platform on which the original model is to be deployed; determine the current weight group to be processed in descending order of the corresponding first model losses; with the current weight group to be processed The minimization of the second model loss corresponding to the reorganization is taken as the optimization goal, and the target quantization result of the current weight group to be processed is searched from the current search range determined based on the target quantization format; and the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small is returned to execute until the target quantization result of each weight group is obtained; wherein, the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range; and the target quantization result of each weight group is used to replace the weight matrix to be quantized of the network layer to be quantized in the original model to obtain the quantized target model.

[0007] Optionally, for each weight group, the target quantization coefficient of the weight group is used to calculate the model loss when the weight group is quantized to the target quantization format according to the activation-perceptual weight quantization algorithm, as the first model loss corresponding to the weight group, including: for each weight group, calculating the average value of the absolute values ​​of each weight in the weight group to obtain the reference quantization coefficient of the weight group; correcting the reference quantization coefficient based on each preset correction coefficient to obtain multiple alternative quantization coefficients; for each alternative quantization coefficient, calculating according to the alternative quantization coefficient, the maximum value of the absolute values ​​of each weight in the weight group, and the target quantization format. An alternative scaling coefficient corresponding to the alternative quantization coefficient; quantizing the weight group to the target quantization format according to the alternative scaling coefficient corresponding to the alternative quantization coefficient, to obtain an alternative quantization result corresponding to the alternative quantization coefficient; dequantizing the corresponding alternative quantization result using the alternative scaling coefficient corresponding to the alternative quantization coefficient to obtain a dequantization result; calculating the model loss corresponding to the alternative quantization coefficient based on the difference between the dequantization result obtained and the weight group; determining the minimum model loss from the calculated model losses as the first model loss when quantizing the weight group to the target quantization format according to the target quantization coefficient of the weight group.

[0008] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1];

[0009] The method of correcting the reference quantization coefficients based on the preset correction coefficients to obtain a plurality of candidate quantization coefficients includes: for each preset correction coefficient, using the preset correction coefficient, correcting the reference quantization coefficient according to a first correction formula to obtain a candidate quantization coefficient corresponding to the preset correction coefficient; wherein the first correction formula is:

[0010] ;

[0011] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0012] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1];

[0013] The method of correcting the reference quantization coefficients based on the preset correction coefficients to obtain multiple candidate quantization coefficients includes: for each preset correction coefficient, using the preset correction coefficient, correcting the reference quantization coefficient according to a second correction formula to obtain the candidate quantization coefficient corresponding to the preset correction coefficient; wherein the second correction formula is:

[0014] ;

[0015] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0016] Optionally, for each candidate quantization coefficient, calculating, according to the candidate quantization coefficient, the maximum value of the absolute values ​​of each weight in the weight group, and the target quantization format, a candidate scaling coefficient corresponding to the candidate quantization coefficient includes:

[0017] For each candidate quantization coefficient, the candidate quantization coefficient, the maximum value of the absolute values ​​of the weights in the weight group, and the target quantization format are used to calculate the candidate scaling factor corresponding to the candidate quantization coefficient according to the scaling factor calculation formula; wherein the scaling factor calculation formula is:

[0018] ;

[0019] Indicates the candidate scaling coefficient corresponding to the candidate quantization coefficient; represents the product of the maximum absolute value of each weight in the weight group and the candidate quantization coefficient; N represents the number of bits indicated by the target quantization format.

[0020] Optionally, the weights in the weight matrix to be quantized are grouped to obtain multiple weight groups, including: determining the weights corresponding to the same input channel of the network layer to be quantized from the weight matrix to be quantized; dividing the determined weights into multiple groups to obtain multiple weight groups; wherein the weights included in a weight group are adjacent in the weight matrix to be quantized.

[0021] Optionally, before searching for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format with the second model loss corresponding to the current weight group to be processed as the optimization goal, the method further includes: obtaining an intermediate quantization result of the current weight group to be processed; wherein the intermediate quantization result is obtained by quantizing the current weight group to be processed to the target quantization format according to the activation-perceptual weight quantization algorithm using the target quantization coefficient of the current weight group to be processed;

[0022] The method takes minimizing the second model loss corresponding to the current weight group to be processed as the optimization goal, and searches for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format, including: determining, within the data range indicated by the target quantization format, a data interval to which the intermediate quantization result of the current weight group to be processed belongs as the current search range; and takes minimizing the second model loss corresponding to the current weight group to be processed as the optimization goal, and searches for the target quantization result of the current weight group to be processed within the current search range.

[0023] Optionally, the optimization objective is to minimize the second model loss corresponding to the current group of weights to be processed, and the target quantization result of the current group of weights to be processed is searched within the current search range, including: taking the minimization of the second model loss corresponding to the current group of weights to be processed as the optimization objective, and searching within the current search range according to a gradient descent method for the target quantization result of the current group of weights to be processed; or taking the minimization of the second model loss corresponding to the current group of weights to be processed as the optimization objective, and searching within the current search range according to a traversal method for the target quantization result of the current group of weights to be processed.

[0024] In a second aspect of an embodiment of the present application, a model quantization device is provided, the device comprising:

[0025] A matrix acquisition module is used to obtain a weight matrix of the network layer to be quantized in a floating point format for each network layer to be quantized in the original model as a weight matrix to be quantized;

[0026] A grouping module, used for grouping the weights in the weight matrix to be quantized to obtain a plurality of weight groups; wherein the weights contained in a weight group correspond to the same input channel of the network layer to be quantized;

[0027] A loss calculation module is used to calculate, for each weight group, a model loss when the weight group is quantized to a target quantization format using a target quantization coefficient of the weight group and according to an activation-perceptual weight quantization algorithm, as a first model loss corresponding to the weight group; wherein, among the candidate quantization coefficients, the model loss when the weight group is quantized to a target quantization format according to the target quantization coefficient of the weight group is the smallest; and the target quantization format is determined based on a quantization format supported by a platform on which the original model is to be deployed;

[0028] A determination module, used to determine the current weight group to be processed according to the order of the corresponding first model loss from large to small;

[0029] The target quantization result acquisition module is used to minimize the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format; and return to execute the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; wherein the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range;

[0030] The replacement module is used to replace the weight matrix to be quantized of the network layer to be quantized in the original model by using the target quantization results of each weight group to obtain a quantized target model.

[0031] Optionally, the loss calculation module is specifically used to: for each weight group, calculate the average value of the absolute values ​​of each weight in the weight group to obtain a reference quantization coefficient of the weight group; correct the reference quantization coefficient based on each preset correction coefficient to obtain a plurality of alternative quantization coefficients; for each alternative quantization coefficient, calculate the alternative scaling coefficient corresponding to the alternative quantization coefficient according to the alternative quantization coefficient, the maximum value of the absolute values ​​of each weight in the weight group, and the target quantization format; quantize the weight group to the target quantization format according to the alternative scaling coefficient corresponding to the alternative quantization coefficient to obtain an alternative quantization result corresponding to the alternative quantization coefficient; dequantize the corresponding alternative quantization result using the alternative scaling coefficient corresponding to the alternative quantization coefficient to obtain a dequantization result; calculate the model loss corresponding to the alternative quantization coefficient based on the difference between the dequantization result obtained and the weight group; determine the minimum model loss from the calculated model losses as the first model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group.

[0032] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1];

[0033] The loss calculation module is specifically used to: for each preset correction coefficient, use the preset correction coefficient to correct the reference quantization coefficient according to a first correction formula to obtain an alternative quantization coefficient corresponding to the preset correction coefficient; wherein the first correction formula is:

[0034] ;

[0035] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0036] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1];

[0037] The loss calculation module is specifically used to: for each preset correction coefficient, use the preset correction coefficient to correct the reference quantization coefficient according to a second correction formula to obtain an alternative quantization coefficient corresponding to the preset correction coefficient; wherein the second correction formula is:

[0038] ;

[0039] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0040] Optionally, the loss calculation module is specifically used to: for each candidate quantization coefficient, use the candidate quantization coefficient, the maximum value of the absolute value of each weight in the weight group, and the target quantization format to calculate the candidate scaling factor corresponding to the candidate quantization coefficient according to the scaling factor calculation formula; wherein the scaling factor calculation formula is:

[0041] ;

[0042] Indicates the candidate scaling coefficient corresponding to the candidate quantization coefficient; represents the product of the maximum absolute value of each weight in the weight group and the candidate quantization coefficient; N represents the number of bits indicated by the target quantization format.

[0043] Optionally, the grouping module is specifically used to: determine the weights corresponding to the same input channel of the network layer to be quantized from the weight matrix to be quantized; divide the determined weights into multiple groups to obtain multiple weight groups; wherein the weights contained in a weight group are adjacent in the weight matrix to be quantized.

[0044] Optionally, the device method further includes: an intermediate quantization result acquisition module, which is used to execute the optimization target of the target quantization result acquisition module to minimize the second model loss corresponding to the weight group to be processed currently, and before searching for the target quantization result of the weight group to be processed currently within the current search range determined based on the target quantization format, to obtain the intermediate quantization result of the weight group to be processed currently; wherein the intermediate quantization result is obtained by quantizing the weight group to be processed currently to the target quantization format according to the activation-perceptual weight quantization algorithm using the target quantization coefficient of the weight group to be processed currently;

[0045] The target quantization result acquisition module is specifically used to: determine, within the data range indicated by the target quantization format, the data interval to which the intermediate quantization result of the current weight group to be processed belongs as the current search range; and search for the target quantization result of the current weight group to be processed within the current search range with the minimum second model loss corresponding to the current weight group to be processed as the optimization goal.

[0046] Optionally, the target quantization result acquisition module is specifically used to: take the minimization of the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed from the current search range in a gradient descent manner; or, take the minimization of the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed from the current search range in a traversal manner.

[0047] In a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for implementing the model quantization method described in any one of the first aspects above when executing the program stored in the memory.

[0048] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the model quantization method described in any one of the first aspects is implemented.

[0049] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described model quantization methods.

[0050] A model quantization method provided by an embodiment of the present application comprises the following steps: for each network layer to be quantized in the original model, an electronic device obtains a weight matrix of the network layer to be quantized in a floating point format as the weight matrix to be quantized; each weight in the weight matrix to be quantized is grouped to obtain a plurality of weight groups; the weights contained in a weight group correspond to the same input channel of the network layer to be quantized; for each weight group, a target quantization coefficient of the weight group is used to calculate a model loss when the weight group is quantized to a target quantization format according to an activation-aware weight quantization algorithm, as the first model loss corresponding to the weight group; among the alternative quantization coefficients, the model loss when the weight group is quantized to a target quantization format according to the target quantization coefficient of the weight group is minimized; the target quantization format is determined based on the quantization format supported by the platform of the original model to be deployed; the current weight to be processed is determined in descending order according to the corresponding first model losses. group; taking the minimum second model loss corresponding to the current weight group to be processed as the optimization goal, searching the target quantization result of the current weight group to be processed from the current search range determined based on the target quantization format; and returning to execute the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found in the current search range; replacing the weight matrix to be quantized of the network layer to be quantized in the original model using the target quantization result of each weight group, to obtain the quantized target model.

[0051] Based on the above processing, the first model loss corresponding to a weight group is: the model loss when the weight group is quantized to the target quantization format. The larger the first model loss corresponding to a weight group, the lower the accuracy of the output data of the model after the weight group is quantized to the target quantization format, that is, the greater the influence of the weight group on the accuracy of the model. Therefore, the electronic device quantizes each weight group in order from large to small according to the corresponding first model loss. And when the electronic device quantizes the current weight group to be processed, it searches from the current search range when the weight group that has a greater impact on the accuracy of the model has been quantized: the target quantization result of the current weight group to be processed that minimizes the corresponding second model loss. That is, for the weight group that has a greater impact on the accuracy of the model, the electronic device will quantize the weight group that has a smaller impact on the accuracy of the model according to the quantization result of the weight group. Then the greater the impact of the weight group on the accuracy of the model, the more times the electronic device performs quantization based on the weight group, and each time the electronic device performs quantization based on the weight group, the quantization result is obtained under the condition of minimizing the model loss. In this way, the quantized target model can be obtained based on the quantization results of each weight group obtained, and the model loss of the target model is small.

[0052] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0054] Figure 1 A first flow chart of the model quantization method provided in the embodiment of the present application;

[0055] Figure 2 A schematic diagram of the weights included in the network layer and the corresponding model loss in the LlaMa model provided in an embodiment of the present application;

[0056] Figure 3 A second flow chart of the model quantization method provided in the embodiment of the present application;

[0057] Figure 4 A third flow chart of the model quantization method provided in an embodiment of the present application;

[0058] Figure 5 A structural diagram of a model quantization device provided in an embodiment of the present application;

[0059] Figure 6A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0061] In the field of deep learning technology, the trained deep learning model can be deployed on the platform, and then the deep learning model can be used on the platform to process business data. However, the model parameters in the trained deep learning model are in Float format. If the platform uses the deep learning model to process business data, a large number of floating-point operations are required, which consumes a large amount of computing resources of the platform. For example, the deep learning model can be a large model built on the basis of the Transformer structure. With the development of large models in recent years, large models have made continuous breakthroughs in processing capabilities for vision, natural language, etc. However, since the weight matrix of the network layer in the large model is in floating-point format, the training process and reasoning process of the large model consume a large amount of computing resources, memory resources, and storage resources. Common large models, such as GPT (Generative pre-trained transformers), OPT (Open Pre-trained Transformer Language Models), LlaMa (Large Language Model MetaArtificial Intelligence), etc., are much larger than the GPU (Graphics Processing Unit) memory limit. For example, storing the GPT3-175B model in Float16 format requires 326GB of memory.

[0062] When running deep learning models with limited computing resources on the platform, problems such as slow computing speed and long response time will occur. To solve the above problems, the model needs to be quantified.

[0063] The embodiment of the present application provides a model quantization method, which is applied to electronic devices. The electronic device can be a server for training deep learning models, a platform for deploying the original model, or other devices that communicate with the server and the platform, and the present application does not limit this. For each network layer to be quantized in the original model, the electronic device can be based on the model quantization method provided by the present application, and in the order from large to small of the corresponding first model loss, quantize each weight group in the floating point format of the network layer to be quantized, and obtain the target quantization result of each weight group; then use the target quantization result of each weight group to replace the weight matrix to be quantized of the network layer to be quantized in the original model, and obtain the quantized target model. The quantized target model is obtained based on the quantization results of each weight group, and the model loss of the target model is small.

[0064] See also Figure 1 , Figure 1 The first flow chart of the model quantization method provided in the embodiment of the present application. The method may include the following steps:

[0065] S101: For each network layer to be quantized in the original model, obtain a weight matrix of the network layer to be quantized in a floating point format as a weight matrix to be quantized.

[0066] S102: Grouping the weights in the weight matrix to be quantized to obtain a plurality of weight groups.

[0067] Among them, the weights contained in a weight group correspond to the same input channel of the network layer to be quantized.

[0068] S103: For each weight group, using the target quantization coefficient of the weight group, according to the activation-perceptual weight quantization algorithm, calculate the model loss when the weight group is quantized to the target quantization format as the first model loss corresponding to the weight group.

[0069] Among the alternative quantization coefficients, the model loss is minimized when the weight group is quantized to a target quantization format according to the target quantization coefficient of the weight group; the target quantization format is determined based on the quantization format supported by the platform of the original model to be deployed.

[0070] S104: Determine the current weight group to be processed according to the order of the corresponding first model losses from large to small.

[0071] S105: Taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, search for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format; and return to execute the step of determining the current weight group to be processed in the order of the corresponding first model losses from large to small, until the target quantization results of each weight group are obtained.

[0072] Among them, the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range.

[0073] S106: Using the target quantization results of each weight group, the weight matrix to be quantized of the network layer to be quantized in the original model is replaced to obtain a quantized target model.

[0074] Based on the model quantization method provided in the embodiment of the present application, the first model loss corresponding to a weight group is: the model loss when the weight group is quantized to the target quantization format. The larger the first model loss corresponding to a weight group, the lower the accuracy of the output data of the model after the weight group is quantized to the target quantization format, that is, the greater the influence of the weight group on the accuracy of the model. Therefore, the electronic device quantizes each weight group in order from large to small in accordance with the corresponding first model loss. And when the electronic device quantizes the current weight group to be processed, it is in the case that the weight group that has a greater impact on the accuracy of the model has been quantized, and searches from the current search range: the target quantization result of the current weight group to be processed that minimizes the corresponding second model loss. That is, for the weight group that has a greater impact on the accuracy of the model, the electronic device will quantize the weight group that has a smaller impact on the accuracy of the model according to the quantization result of the weight group. Then the greater the impact of the weight group on the accuracy of the model, the more times the electronic device performs quantization based on the weight group, and each time the electronic device performs quantization based on the weight group, the quantization result is obtained under the condition of minimizing the model loss. In this way, the quantized target model can be obtained based on the quantization results of each weight group obtained, and the model loss of the target model is small.

[0075] For step S101, the original model is a deep learning model including multiple network layers, and among the multiple network layers, there is a network layer (hereinafter referred to as the candidate network layer) in which each weight in the weight matrix is ​​in floating point format. For example, the original model may be an SSD (Single Shot MultiBox Detector) model for target detection, an LSTM (Long Short-Term Memory) model for text recognition, an LLM for answering questions carried in input text, etc.; the floating point format may be Float16, Float32, etc.

[0076] Based on the hardware performance of the platform on which the original model is to be deployed, the network layer to be quantized among the multiple network layers of the original model can be determined. For example, there can be multiple candidate network layers in the original model.

[0077] The hardware performance of an electronic device can be determined by comprehensively considering the memory size, CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. of the electronic device. For example, when the CPU clock frequency is high, the cache is large, and the bus frequency is high, the CPU performance can be considered to be good; when the GPU cache is large, the GPU performance can also be considered to be good. Correspondingly, when the platform has a large memory, a good CPU performance, and a good GPU performance, the platform's hardware performance can be considered to be good. When the platform's hardware performance is good, some of the candidate network layers can be used as network layers to be quantized; or, the multiple candidate network layers can all be used as network layers to be quantized. When the platform's hardware performance is relatively poor, such as when the platform's CPU performance is poor, in order to run the quantized target model on the platform, the multiple candidate network layers can all be used as network layers to be quantized. For each network layer to be quantized, the weight matrix of the network layer to be quantized in floating-point format is the weight matrix to be quantized.

[0078] Exemplarily, when the original model is an SSD model for target detection, the input data of the original model may be an image to be detected, and the output data may be a detection result for the image to be detected, such as a detection result that may indicate the position of a specified object in the image to be detected. The network layer to be quantized may be a convolutional layer for feature extraction in SSD, and the input data of the convolutional layer may be a pixel matrix recording the pixel values ​​of each pixel point of the image to be detected, and the output data may be a feature matrix extracted based on the pixel matrix.

[0079] Exemplarily, when the original model is an LLM for answering questions carried in an input text, the input data of the original model may be the input text carrying the question, and the output data may be the answer text for the question. The network layer to be quantized may be a pooling layer for downsampling processing, the input data of the pooling layer may be a feature matrix extracted based on the input text, and the output data may be a feature matrix obtained by downsampling the feature matrix.

[0080] For step S102 and step S103, for each network layer to be quantized, the weight matrix to be quantized of the network layer to be quantized includes a plurality of weights to be quantized in floating point format. A column of weights in the weight matrix to be quantized corresponds to an input channel of the network layer to be quantized; a row of weights corresponds to an output channel of the network layer to be quantized.

[0081] It is understandable that the model loss of the model obtained by quantizing the weight matrix to be quantized to different quantization formats is different. For example, the model loss when the weight matrix to be quantized is quantized to Int (integer) 8 format is smaller than the model loss when the weight matrix to be quantized is quantized to Int2 format. In addition, the model operation efficiency of the model obtained by quantizing the weight matrix to be quantized to different quantization formats is also different. For example, the model operation efficiency when the weight matrix to be quantized is quantized to Int8 format is lower than the model operation efficiency when the weight matrix to be quantized is quantized to Int2 format. Therefore, the model loss requirements and model operation efficiency requirements of the target model to be obtained can be weighed, and the target quantization format to which the weight group needs to be quantized can be determined from the quantization formats supported by the platform on which the original model is to be deployed. For example, the quantization formats supported by the platform may include: Int8, Int4, and Int2. When a target model with lower model loss is required but high model operation efficiency is not required, the target quantization format may be Int8. When a target model with higher model operation efficiency is required but lower model loss is not required, the target quantization format may be Int2. When a target model with both low model loss and high model operation efficiency is required, the target quantization format may be Int4.

[0082] For each weight, the difference between the output data obtained by the quantized network layer corresponding to the weight and the network layer to be quantized for the same input data can be used as the model loss corresponding to the weight. The quantized network layer corresponding to a weight is: a network layer obtained by quantizing the weight to the target quantization format and not quantizing other weights. Obviously, the model loss corresponding to a weight can indicate the impact of the weight on the accuracy of the model. Experimental results show that different weights have different effects on the accuracy of the model. The weights that have a greater impact on the accuracy of the model, that is, the weights with a larger corresponding model loss, can be called sensitive weights; correspondingly, the weights with a smaller corresponding model loss can be called redundant weights. Obviously, if the model loss of the quantized target model is to be smaller, it is necessary to fully quantize the weights based on the corresponding larger model loss.

[0083] For example, see Figure 2 , Figure 2 A schematic diagram of the weights included in the network layer in the LlaMa model provided in an embodiment of the present application and the corresponding model loss. Figure 2 The coordinate values ​​of the horizontal axis in correspond to the input channels of the network layer; the coordinate values ​​of the vertical axis correspond to the output channels of the network layer. Figure 2 The value of a point in represents the model loss corresponding to the weight indicated by the point. The weight indicated by the point is: the weight indicated by the input channel represented by the horizontal coordinate of the point and the output channel represented by the vertical coordinate of the point. Figure 2 The model loss corresponding to the weight indicated by the point through which the vertical line passes is larger.

[0084] based on Figure 2 It can be seen that a network layer may have tens of thousands of output channels and thousands of input channels, that is, the number of weights in the weight matrix to be quantized is extremely large. In a weight matrix to be quantized, only a small number of weights correspond to a large model loss. In addition, the model losses corresponding to the weights corresponding to some input channels in the network layer are large; the differences in the model losses corresponding to the weights corresponding to the same input channel may be large.

[0085] Obviously, if the electronic device subsequently performs quantization in units of weights, it will need to be quantized tens of millions of times, resulting in the quantization of the network layer to be quantized taking too long, that is, the quantization efficiency is too low. Therefore, in order to improve the quantization efficiency of the network layer to be quantized, the electronic device can divide the weight matrix to be quantized into multiple weight groups based on the columns of the weight matrix to be quantized.

[0086] For each weight group, the difference between the output data obtained by the quantization network layer corresponding to the weight group and the network layer to be quantized for the same input data can be used as the model loss corresponding to the weight group (that is, the first model loss corresponding to the weight group in the subsequent embodiments). The quantization network layer corresponding to a weight group is: a network layer obtained by quantizing each weight in the weight group to the target quantization format and not quantizing other weight groups. Obviously, the model loss corresponding to a weight group can indicate the impact of the weight group on the accuracy of the model, and when a weight group includes weights with a larger corresponding model loss, the first model loss corresponding to the weight group is also larger. Subsequently, the electronic device quantizes each weight group in sequence according to the first model loss corresponding to each weight group from large to small, taking the weight group as a unit.

[0087] It can be understood that the more weight groups there are, the fewer weights included in each weight group obtained by division, and the closer the model losses corresponding to the weights in a weight group are. When the weights in the weight group are uniformly quantized, the interference caused by the weights with larger corresponding model losses can be reduced, so that the model loss of the target model obtained later is smaller. The fewer the number of weight groups, the fewer times the network layer to be quantized is quantized, making the efficiency of quantizing the original model higher. Moreover, when the hardware performance of the electronic device is good, even if the number of weight groups obtained by division is large, the efficiency of quantizing the original model can be higher.

[0088] Therefore, it is necessary to weigh the model loss requirements of the target model, the quantization efficiency requirements for the original model, and the hardware performance of the electronic device to jointly determine the number of weight groups obtained by division. If you need to obtain a target model with smaller model loss, the hardware performance of the electronic device is better, and you do not require higher quantization efficiency for the original model, you can set the division to obtain a larger number of weight groups; when you need to improve the efficiency of obtaining the target model, if the hardware performance of the electronic device is not high, and the requirements for the model loss of the target model are not high, you can set the division to obtain a smaller number of weight groups. For the way in which the electronic device obtains multiple weight groups, please refer to the detailed description of the subsequent embodiments, which will not be repeated here.

[0089] For each weight group, the electronic device can determine the target quantization coefficient of the weight group from the candidate quantization coefficients based on the AWQ (Activation-aware WeightQuantization) algorithm. Then, the model loss when the weight group is quantized to the target quantization format is calculated according to the AWQ algorithm to obtain the first model loss corresponding to the weight group.

[0090] For step S104 and step S105, in order to fully quantify the weights with larger corresponding model losses, the electronic device can quantize each weight group in sequence according to the corresponding first model loss from large to small, and the current quantized weight group is the current weight group to be processed.

[0091] Based on the target quantization format, the electronic device can determine the current search range. It is understandable that the larger the current search range, the larger the amount of data to be processed when the electronic device searches for the target quantization result of the current weight group to be processed within the current search range, and the longer it takes for the electronic device to determine the target quantization result of the current weight group to be processed, and the lower the quantization efficiency for the original model; the higher the accuracy of the target quantization result of the current weight group to be processed, the smaller the model loss of the target model obtained subsequently. The higher the hardware performance of the electronic device, the shorter the time it takes for the electronic device to search for the target quantization result of the current weight group to be processed.

[0092] Therefore, the model loss requirements of the target model to be obtained, the quantization efficiency requirements for the original model, and the hardware performance of the electronic device can be weighed to determine the current search range. If a target model with a smaller model loss is required, the hardware performance of the electronic device is higher, and the quantization efficiency requirements for the original model are not high, the data range indicated by the target quantization format can be used as the current search range. For example, when the target quantization format is Int8, the current search range can be (-128, 127). When the hardware performance of the electronic device is not high, it is necessary to improve the quantization efficiency for the original model, and the model loss of the target model is not required to be smaller, the electronic device can also determine a data interval from the data range as the current search range. The way in which the electronic device determines the data interval is described in detail in the subsequent embodiments and will not be repeated here.

[0093] After determining the current weight group to be processed, the electronic device obtains the current reference weight matrix using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range.

[0094] For example, the electronic device can first use the currently determined target quantization result to replace the corresponding weight group in the weight matrix to be quantized to obtain the current weight matrix to be replaced. Then, the search is performed within the current search range. A data combination is found, that is, a quantization result of the current weight group to be processed is found within the current search range. The current weight group to be processed in the current weight matrix to be replaced is replaced by the quantization result, and a current reference weight matrix is ​​obtained. The difference between the calculation results of the current reference weight matrix and the weight matrix to be quantized for the same input data can be used as the second model loss corresponding to the current reference weight matrix, that is, a second model loss corresponding to the current weight group to be processed.

[0095] Within the current search range, the electronic device can find multiple data combinations, that is, find multiple quantization results. For each quantization result, the electronic device can calculate a second model loss based on the quantization result in the above manner. The corresponding quantization result with the smallest second model loss is the target quantization result of the current weight group to be processed. Then, the electronic device continues to determine the current weight group to be processed in the order of the corresponding first model loss from large to small. This cycle is repeated until the target quantization results of each weight group are obtained.

[0096] That is, for each quantization result found by the electronic device within the current search range, the electronic device uses the quantization result and the determined target quantization result to jointly obtain the current reference weight matrix corresponding to the quantization result. Subsequently, based on the second model loss corresponding to each current reference weight matrix, the target quantization result of the current weight group to be processed is determined from each found quantization result.

[0097] For example, in the order of the corresponding first model loss from large to small, the electronic device uses the corresponding weight group with the largest first model loss (hereinafter referred to as the first weight group) as the current weight group to be processed. Obviously, there is no currently determined target quantization result at this time, then the current weight matrix to be replaced is the weight matrix to be quantized. After the electronic device searches for the target quantization result of the first weight group within the current search range, it uses the corresponding weight group with the second largest first model loss (hereinafter referred to as the second weight group) as the current weight group to be processed.

[0098] At this time, the electronic device has determined the target quantization result of the first weight group, and the current weight matrix to be replaced is obtained by replacing the first weight group in the weight matrix to be quantized using the target quantization result of the first weight group. After the electronic device searches for the target quantization result of the second weight group within the current search range, it uses the corresponding weight group with the third largest loss in the first model (hereinafter referred to as the third weight group) as the current weight group to be processed.

[0099] At this point, the electronic device has determined the target quantization result of the first weight group and the target quantization result of the second weight group, and the current weight matrix to be replaced is obtained by replacing the first weight group and the second weight group in the weight matrix to be quantized, respectively, using the target quantization result of the first weight group and the target quantization result of the second weight group. After the electronic device searches for the target quantization result of the third weight group within the current search range, it uses the corresponding first model loss fourth largest weight group as the current weight group to be processed.

[0100] By analogy, the electronic device quantizes each weight group in sequence according to the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained.

[0101] For step S106, the electronic device can use the target quantization results of each weight group to replace the corresponding weight group in the weight matrix to be quantized, that is, quantize the network layer to be quantized to obtain the quantized network layer. Then it includes the model of the quantized network layer, that is, the quantized target model. When there are multiple network layers to be quantized, all of the multiple network layers to be quantized are quantized to obtain the quantized target model.

[0102] In some embodiments, the above step S102 may include the following steps:

[0103] Step 1: From the weight matrix to be quantized, determine the weights corresponding to the same input channel of the network layer to be quantized.

[0104] Step 2: Divide the determined weights into multiple groups to obtain multiple weight groups.

[0105] Among them, the weights contained in a weight group are adjacent in the weight matrix to be quantized.

[0106] As mentioned above, the model losses corresponding to the weights corresponding to some input channels in the network layer to be quantized are relatively large; the differences in the model losses corresponding to the weights corresponding to the same input channel may be relatively large. That is, the weights with relatively large model losses in the weight matrix to be quantized are adjacent to each other in the columns of the matrix to be quantized. If the weights corresponding to the same output channel of the network layer to be quantized are divided, that is, the weights are divided according to the weights of each row in the weight matrix to be quantized, then the differences in the model losses corresponding to the weights in the weight group obtained by the division will be relatively large. When quantization is performed in units of weight groups, the weights with relatively large differences in the corresponding model losses in the weight group are uniformly quantized, resulting in poor quantization of the weights with relatively large model losses in the weight group, and then resulting in relatively large model losses of the target model obtained subsequently. Therefore, the electronic device will determine the weights corresponding to the same input channel of the network layer to be quantized from the weight matrix to be quantized, that is, the weights are divided according to the weights of each column in the weight matrix to be quantized, so that the differences in the model losses corresponding to the weights in the weight group obtained by the division are relatively small. For the weight group that includes weights with larger corresponding model losses, the interference caused by the weights with smaller corresponding model losses will be reduced, so that the quantization effect of the weights with larger corresponding model losses is better, which can reduce the model loss of the subsequent target model.

[0107] In one implementation, the electronic device may first estimate the estimated number of weight groups that need to be obtained based on the model loss requirements of the target model, the quantization efficiency requirements for the original model, and the hardware performance of the electronic device; and then, according to the determined number of weights corresponding to the same input channel of the network layer to be quantized and the number of input channels of the network layer to be quantized, determine the method of dividing the determined weights when the determined weights are evenly divided into multiple weight groups and the total number of weight groups finally obtained by the division is closest to the estimated number.

[0108] Exemplarily, the estimated number is 800. For the weight matrix to be quantized of size 100×200, the weight matrix to be quantized can be divided into 200 columns according to the input channels of the network layer to be quantized, each column including 100 weights, that is, the 100 weights corresponding to each of the 200 input channels of the network layer to be quantized are determined. Then each column of weights needs to be divided into 4 weight groups, that is, each weight group includes 25 weights. Therefore, for each column of weights, the 1st to 25th weights in the column of weights are divided into a weight group, the 26th to 50th weights are divided into a weight group, and so on. The weight matrix to be quantized is divided into 800 weight groups, each weight group includes 25 weights, and the weights included in each weight group are adjacent in the weight matrix to be quantized, and all correspond to the same input channel of the network layer to be quantized.

[0109] In some embodiments, Figure 1 Based on Figure 3 , the above step S103 may include the following steps:

[0110] S1031: For each weight group, calculate the average value of the absolute values ​​of the weights in the weight group to obtain a reference quantization coefficient of the weight group.

[0111] S1032: Correct the reference quantization coefficients based on the preset correction coefficients to obtain multiple candidate quantization coefficients.

[0112] S1033: For each candidate quantization coefficient, calculate a candidate scaling coefficient corresponding to the candidate quantization coefficient according to the candidate quantization coefficient, the maximum value of the absolute values ​​of the weights in the weight group, and the target quantization format.

[0113] S1034: quantizing the weight group to a target quantization format according to the candidate scaling coefficient corresponding to the candidate quantization coefficient, and obtaining a candidate quantization result corresponding to the candidate quantization coefficient.

[0114] S1035: Dequantize the corresponding candidate quantization result using the candidate scaling coefficient corresponding to the candidate quantization coefficient to obtain a dequantization result.

[0115] S1036: Based on the difference between the obtained inverse quantization result and the weight group, calculate the model loss corresponding to the alternative quantization coefficient.

[0116] S1037: Determine the minimum model loss from the calculated model losses as the first model loss when quantizing the weight group to the target quantization format according to the target quantization coefficient of the weight group.

[0117] For each weight group, the reference quantization coefficient of the weight group is obtained based on all weights included in the weight group, that is, the reference quantization coefficient can indicate each weight included in the weight group.

[0118] In some embodiments, each preset correction coefficient is: the value of each average point in the data interval [0, 1]; for example, the data interval [0, 1] may include 20 average points, and the values ​​of the 20 average points may include: 0.05, 0.1, 0.15, ..., 1. Subsequently, the target quantization coefficient of the weight group is searched in the plurality of preset correction coefficients based on a grid search method.

[0119] In one implementation, the above step S1032 may include the following steps: for each preset correction coefficient, using the preset correction coefficient, correcting the reference quantization coefficient according to a first correction formula to obtain an alternative quantization coefficient corresponding to the preset correction coefficient; wherein the first correction formula is:

[0120] ;

[0121] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0122] For each preset correction factor , the electronic device will calculate the preset correction coefficient according to the first correction formula above Corresponding candidate quantization coefficients. For example, the reference quantization coefficients is 5, then based on the above example, the preset correction coefficient When is 0.05, the alternative quantization coefficient corresponding to 0.05 is ; Preset correction factor When is 0.1, the alternative quantization coefficient corresponding to 0.1 is ; Preset correction factor When is 0.15, the alternative quantization coefficient corresponding to 0.15 is By analogy, the electronic device obtains each preset correction coefficient based on the above first correction formula: The corresponding candidate quantization coefficients.

[0123] In another implementation, the above step S1032 may include the following steps: for each preset correction coefficient, using the preset correction coefficient, correcting the reference quantization coefficient according to a second correction formula to obtain an alternative quantization coefficient corresponding to the preset correction coefficient; wherein the second correction formula is:

[0124] ;

[0125] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0126] For each preset correction factor , the electronic device will calculate the preset correction coefficient according to the second correction formula above Corresponding candidate quantization coefficients. For example, the reference quantization coefficients is 10, then based on the above example, the preset correction coefficient When it is 0.05, the alternative quantization coefficient corresponding to 0.05 is 0.05×10, that is, 0.5; the preset correction coefficient When it is 0.1, the alternative quantization coefficient corresponding to 0.1 is 0.1×10, that is, 1; the preset correction coefficient When 0.15, the candidate quantization coefficient corresponding to 0.15 is 0.15×10, that is, 1.5. Similarly, the electronic device obtains each preset correction coefficient based on the above second correction formula: The corresponding candidate quantization coefficients.

[0127] It is understandable that no matter which of the above methods is used to obtain the candidate quantization coefficients, the candidate quantization coefficients are all obtained in the same way. For example, the candidate quantization coefficients are all obtained based on the above first correction formula; or, the candidate quantization coefficients are all obtained based on the above second correction formula.

[0128] Based on the above processing, for each weight group, the reference quantization coefficient of the weight group is: the average value of the absolute values ​​of each weight in the weight group, that is, the reference quantization coefficient of a weight group can measure the average size of each weight included in the weight group. Then the electronic device corrects the reference quantization coefficient based on each preset correction coefficient, and then determines the target quantization coefficient of the weight group from the corrected alternative quantization coefficients, and then quantizes each weight included in the weight group based on the target quantization coefficient. That is, the weights included in a weight group are comprehensively determined to jointly determine the unified quantization of the weight group, which can reduce the interference caused by the corresponding weights with larger model loss and reduce the model loss of the target model obtained subsequently.

[0129] In some embodiments, the above step S1033 may include the following steps: for each candidate quantization coefficient, using the candidate quantization coefficient, the maximum value of the absolute values ​​of each weight in the weight group, and the target quantization format, calculate the candidate scaling factor corresponding to the candidate quantization coefficient according to the scaling factor calculation formula; wherein the scaling factor calculation formula is:

[0130] ;

[0131] Indicates the candidate scaling coefficient corresponding to the candidate quantization coefficient; It represents the product of the maximum absolute value of each weight in the weight group and the candidate quantization coefficient; N represents the number of bits indicated by the target quantization format.

[0132] For example, when the target quantization format is Int8, N represents 8; when the target quantization format is Int2, N represents 2.

[0133] According to the alternative scaling factor, the electronic device may quantize each weight in the weight group according to the following quantization formula:

[0134] ;

[0135] in, It represents the alternative quantization result obtained by quantizing the weight group w based on the alternative quantization coefficient s; Round represents rounding processing.

[0136] Using the alternative scaling coefficients, the electronic device can perform inverse quantization on the alternative quantization results of the weight group according to the following inverse quantization formula to obtain an inverse quantization result: :

[0137] ;

[0138] In this way, for the weight group, the electronic device can obtain the inverse quantization result of the weight group calculated based on the candidate quantization coefficient.

[0139] Then, the electronic device may calculate the model loss corresponding to the candidate quantization coefficient according to the calculated dequantization result of the weight group and the following loss formula:

[0140] ;

[0141] in, Indicates the model loss corresponding to the alternative quantization coefficient; Represents the loss value calculated based on the preset loss function; Represents the input data of the network layer to be quantized that is currently quantized. For example, the preset loss function may be MSE (Mean Square Error), cross entropy loss function, etc.

[0142] The model loss corresponding to an alternative quantization coefficient indicates: the difference in output data obtained by the quantization network layer corresponding to the alternative quantization coefficient and the network layer to be quantized for the same input data. The quantization network layer corresponding to an alternative quantization coefficient is: the network layer obtained by quantizing each weight in the weight group to the target quantization format according to the alternative quantization coefficient.

[0143] After calculating the model losses corresponding to the candidate quantization coefficients, the electronic device can determine the minimum model loss (i.e., the first model loss) from the calculated model losses. The candidate quantization coefficient corresponding to the first model loss is the target quantization coefficient used when quantizing the weight group when the loss of the model obtained by quantizing the weight group is the minimum.

[0144] Subsequently, the electronic device quantizes each weight group in order from large to small according to the first model loss. For the corresponding weight group with a large first model loss, the electronic device will quantize the corresponding weight group with a small first model loss according to the quantization result of the weight group. The greater the first model loss corresponding to the weight group, the more times the electronic device performs quantization based on the weight group. That is, in the quantization process of the network layer to be quantized, the corresponding weight group with a large first model loss will be fully utilized, that is, the weights with a large corresponding model loss will be fully quantized to reduce the model loss of the target model obtained by quantization.

[0145] In some embodiments, Figure 1 Based on Figure 4 Before the above step S105, the method may further include the following steps:

[0146] S107: Obtain the intermediate quantization result of the weight group to be processed currently.

[0147] The intermediate quantization result is obtained by quantizing the current weight group to be processed to the target quantization format according to the activation-aware weight quantization algorithm using the target quantization coefficient of the current weight group to be processed.

[0148] Accordingly, step S105 may include the following steps:

[0149] S1051: Determine, within the data range indicated by the target quantization format, a data interval to which the intermediate quantization result of the weight group to be processed currently belongs as the current search range.

[0150] S1052: Taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, search for the target quantization result of the current weight group to be processed within the current search range; and return to execute the step of determining the current weight group to be processed in the order of the corresponding first model losses from large to small, until the target quantization results of each weight group are obtained.

[0151] As mentioned above, for each weight group, the electronic device quantizes the weight group based on each candidate quantization coefficient, and then determines the target quantization coefficient of the weight group from each candidate quantization coefficient based on each quantization result. Therefore, when the electronic device determines the target quantization coefficient of the weight group, it has already obtained the quantization result (i.e., the intermediate quantization result) of quantizing the weight group based on the target quantization coefficient. This process can be called a process of determining the intermediate quantization result based on the nearest neighbor rounding quantization method.

[0152] Then, within the data range indicated by the target quantization format (hereinafter referred to as the target data range), determine the data interval to which the intermediate quantization result belongs, that is, determine the current search range. Search for the target quantization result within the current search range, that is, search for the target quantization result on integers adjacent to the intermediate quantization result.

[0153] In one implementation, the target data range can be pre-divided into multiple data intervals according to the hardware performance of the electronic device, and then the data interval to which the intermediate quantization result belongs is determined from the multiple data intervals. For example, when the hardware performance of the electronic device is high, the number of data intervals obtained by division is small, and each data interval includes more data; when the hardware performance of the electronic device is not high, the number of data intervals obtained by division is large, and each data interval includes less data.

[0154] In one implementation, the specified length can be determined according to the hardware performance of the electronic device. Within the target data range, a data interval with the intermediate quantization result as the data center and the interval length as the specified length is determined, that is, the data interval to which the intermediate quantization result belongs is determined. When the hardware performance of the electronic device is high, the specified length is large, that is, the data interval with the interval length of the specified length includes more data; when the hardware performance of the electronic device is not high, the specified length is small, that is, the data interval with the interval length of the specified length includes less data.

[0155] It can be understood that a weight group includes multiple weights, and the intermediate quantization result of a weight group includes: the intermediate quantization result of each weight included in the weight group. For each weight in the weight group currently to be processed, the electronic device can determine the data interval to which the intermediate quantization result of the weight belongs according to the intermediate quantization result of the weight, as the current search range of the weight. Subsequently, the target quantization result of the weight is searched within the current search range of the weight. The target quantization result of each weight in the weight group currently to be processed is the target quantization result of the weight group currently to be processed that is found.

[0156] In some embodiments, the above-mentioned step S1052 may include the following steps: taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, searching for the target quantization result of the current weight group to be processed from the current search range in a gradient descent manner; or, taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, searching for the target quantization result of the current weight group to be processed from the current search range in a traversal manner.

[0157] After determining the current search range, the electronic device can determine multiple data combinations, that is, multiple quantization results, in the current search range in a gradient descent manner or a traversal manner. For each quantization result, the electronic device can calculate the second model loss corresponding to the quantization result. The quantization result with the smallest corresponding second model loss is the target quantization result of the current weight group to be processed. That is, the electronic device searches for the target quantization result on adjacent integers.

[0158] Exemplarily, the electronic device divides the weight matrix to be quantized into N weight groups, each weight group including K weights. Based on the method in the above embodiment, the electronic device can determine the first model loss corresponding to each weight group. The first model loss corresponding to a weight group can be recorded as , the weight group can be expressed as i=1, 2, ..., N, i represents the position of the first model loss corresponding to each weight group in the order from large to small. For example, the first model loss corresponding to the weight group with the largest first model loss (i.e., the first weight group in the above embodiment) is recorded as , the first weight group is recorded as .

[0159] Based on the order of the first model losses corresponding to each weight group from large to small, the quantization order of each weight group determined by the electronic device is recorded as: It is understandable that the electronic device only needs to determine the quantization order of each weight group, and does not require to rearrange each weight group in the weight matrix to be quantized.

[0160] When the current weight group to be processed is the first weight group, the electronic device can determine the target quantization result of the first weight group based on the following formula (1):

[0161] …… (1)

[0162] in, represents the quantization result of the first weight group found in the current search range; Indicates the second model loss corresponding to the quantized result found within the current search range for the first weight group; represents the loss value calculated based on the preset loss function; x represents the input data of the network layer to be quantized currently; Indicates that the target quantization result of the first weight group is determined when the corresponding second model loss is minimized .

[0163] Then, the electronic device determines the corresponding weight group with the second largest loss in the first model (i.e., the second weight group in the aforementioned embodiment) as the weight group to be processed currently, and determines the target quantization result of the second weight group based on the following formula (2):

[0164] …… (2)

[0165] in, represents the quantization result of the second weight group found within the current search range; represents the target quantization result based on the determined first weight group, and the second model loss corresponding to the quantization result found within the current search range for the second weight group; Indicates that when the target quantization result of the first weight group has been determined and the corresponding second model loss is minimal, the target quantization result of the second weight group is determined .

[0166] By analogy, the electronic device quantizes each weight group in order from large to small according to the corresponding first model loss until the target quantization result of each weight group is obtained. The formulas used by the electronic device to determine the target quantization result of each weight group (including the above formula (1) and formula (2)) can be collectively referred to as group loss functions. Since the electronic device subsequently determines the target quantization result of the corresponding weight group with smaller first model loss based on the determined target quantization result, that is, fully reduces the group loss function corresponding to the weight group with larger model loss, the model loss of the target model finally quantized is smaller.

[0167] It can be seen that the electronic device performs N quantizations based on the first weight group, performs N-1 quantizations based on the second weight group, and so on, and performs quantization once based on the corresponding weight group with the smallest first model loss. In this way, by quantizing the distribution of each weight group, the target quantization result of each weight group is determined based on: the target quantization result of other weight groups whose corresponding first model loss is greater than the first model loss corresponding to the weight group. This quantization method can be called a method of performing model quantization using the high correlation between parameters within the layer. In the quantization process of the network layer to be quantized, the corresponding weight group with large first model loss will be fully utilized, that is, the weights with large corresponding model losses in these weight groups will be fully utilized to reduce the model loss of the target model obtained by the final quantization.

[0168] Compared with the method of directly quantizing each weight group separately in the prior art, although the weights in the weight matrix to be quantized are quantized to the target quantization format, the corresponding weights with larger model loss and the corresponding weights with smaller model loss are quantized in the same way, that is, the corresponding weights with larger model loss will not be fully quantized, resulting in a large model loss of the quantized model obtained based on the prior art. However, the method provided in the present application is fully quantized based on the corresponding weights with larger model loss, so that the model loss of the target model finally quantized is smaller, that is, the model loss of the target model obtained based on the present application is lower than the model loss of the quantized model obtained based on the prior art. Therefore, when low-bit quantization is performed, that is, when the target quantization format is an integer format with a smaller number of bits such as Int2 and Int4, the model loss of the quantized model obtained based on the prior art is large, and the accuracy of the result of processing business data based on the quantized model is not high, resulting in the existing model quantization method not being suitable for low-bit quantization. Based on the model quantization method provided in the embodiment of the present application, a target model with lower model loss can be obtained, that is, the accuracy of the result of processing business data by the target model obtained in this way is higher, and the model quantization method provided in the embodiment of the present application is suitable for low-bit quantization.

[0169] In addition, when the quantized target model is used to process business data, since the weight matrix of the network layer in the target model is in the target quantization format, fewer computing resources are consumed when running the target model, that is, the hardware performance requirements of the platform are lower. On platforms with lower hardware performance, such as RTX3080 (Ray Tracing Texel eXtreme3080), T4 (Tier4), end-side devices, etc., the target model can also be supported to run, that is, the target model can be run on platforms with lower hardware performance, which can reduce the hardware performance requirements for using the target model and reduce the cost of using the target model.

[0170] In addition, since the amount of data involved in integer operations is much smaller than the amount of data involved in floating-point operations, the quantized target model has a faster computing speed than the original model, and the original model is compressed, reducing the memory occupied by the target model.

[0171] Based on the same inventive concept as the above-mentioned model quantization method, the present application also provides a model quantization device. Figure 5 , Figure 5 A structural diagram of a model quantization device provided in an embodiment of the present application, the device includes:

[0172] The matrix acquisition module 501 is used to acquire, for each network layer to be quantized in the original model, a weight matrix of the network layer to be quantized in a floating point format as a weight matrix to be quantized;

[0173] A grouping module 502 is used to group the weights in the weight matrix to be quantized to obtain multiple weight groups; wherein the weights contained in a weight group correspond to the same input channel of the network layer to be quantized;

[0174] The loss calculation module 503 is used to calculate, for each weight group, the model loss when the weight group is quantized to the target quantization format according to the activation-perceptual weight quantization algorithm using the target quantization coefficient of the weight group, as the first model loss corresponding to the weight group; wherein, among the candidate quantization coefficients, the model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group is the smallest; the target quantization format is determined based on the quantization format supported by the platform on which the original model is to be deployed;

[0175] A determination module 504 is used to determine the current weight group to be processed according to the order of the corresponding first model losses from large to small;

[0176] The target quantization result acquisition module 505 is used to take the minimum second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format; and return to execute the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; wherein the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range;

[0177] The replacement module 506 is used to replace the weight matrix to be quantized of the network layer to be quantized in the original model by using the target quantization results of each weight group to obtain a quantized target model.

[0178] Optionally, the loss calculation module 503 is specifically used to: for each weight group, calculate the average value of the absolute values ​​of each weight in the weight group to obtain a reference quantization coefficient of the weight group; correct the reference quantization coefficient based on each preset correction coefficient to obtain a plurality of alternative quantization coefficients; for each alternative quantization coefficient, calculate the alternative scaling coefficient corresponding to the alternative quantization coefficient according to the alternative quantization coefficient, the maximum value of the absolute values ​​of each weight in the weight group, and the target quantization format; quantize the weight group to the target quantization format according to the alternative scaling coefficient corresponding to the alternative quantization coefficient to obtain an alternative quantization result corresponding to the alternative quantization coefficient; dequantize the corresponding alternative quantization result using the alternative scaling coefficient corresponding to the alternative quantization coefficient to obtain a dequantization result; calculate the model loss corresponding to the alternative quantization coefficient based on the difference between the dequantization result obtained and the weight group; determine the minimum model loss from the calculated model losses as the first model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group.

[0179] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1];

[0180] The loss calculation module 503 is specifically used to: for each preset correction coefficient, use the preset correction coefficient to correct the reference quantization coefficient according to a first correction formula to obtain an alternative quantization coefficient corresponding to the preset correction coefficient; wherein the first correction formula is:

[0181] ;

[0182] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0183] Optionally, each preset correction coefficient is: the value of each average point in the data interval [0, 1]; the loss calculation module 503 is specifically used to: for each preset correction coefficient, use the preset correction coefficient to correct the reference quantization coefficient according to the second correction formula to obtain the candidate quantization coefficient corresponding to the preset correction coefficient; wherein the second correction formula is:

[0184] ;

[0185] s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

[0186] Optionally, the loss calculation module 503 is specifically used to: for each candidate quantization coefficient, use the candidate quantization coefficient, the maximum value of the absolute value of each weight in the weight group, and the target quantization format, and calculate the candidate scaling factor corresponding to the candidate quantization coefficient according to the scaling factor calculation formula; wherein the scaling factor calculation formula is:

[0187] ;

[0188] Indicates the candidate scaling coefficient corresponding to the candidate quantization coefficient; represents the product of the maximum absolute value of each weight in the weight group and the candidate quantization coefficient; N represents the number of bits indicated by the target quantization format.

[0189] Optionally, the grouping module 502 is specifically used to: determine the weights corresponding to the same input channel of the network layer to be quantized from the weight matrix to be quantized; divide the determined weights into multiple groups to obtain multiple weight groups; wherein the weights contained in a weight group are adjacent in the weight matrix to be quantized.

[0190] Optionally, the device method further includes: an intermediate quantization result acquisition module, which is used to execute the optimization target of minimizing the second model loss corresponding to the current weight group to be processed in the target quantization result acquisition module 505, and before searching for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format, to obtain the intermediate quantization result of the current weight group to be processed; wherein the intermediate quantization result is obtained by quantizing the current weight group to be processed to the target quantization format according to the activation perception weight quantization algorithm using the target quantization coefficient of the current weight group to be processed;

[0191] The target quantization result acquisition module 505 is specifically used to: determine, within the data range indicated by the target quantization format, the data interval to which the intermediate quantization result of the current weight group to be processed belongs as the current search range; and search for the target quantization result of the current weight group to be processed within the current search range with the minimum second model loss corresponding to the current weight group to be processed as the optimization goal.

[0192] Optionally, the target quantization result acquisition module 505 is specifically used to: take the minimization of the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed from the current search range in a gradient descent manner; or, take the minimization of the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed from the current search range in a traversal manner.

[0193] Based on the model quantization device provided in the embodiment of the present application, the first model loss corresponding to a weight group is: the model loss when the weight group is quantized to the target quantization format. The larger the first model loss corresponding to a weight group, the lower the accuracy of the output data of the model after the weight group is quantized to the target quantization format, that is, the greater the influence of the weight group on the accuracy of the model. Therefore, the electronic device quantizes each weight group in order from large to small in accordance with the corresponding first model loss. And when the electronic device quantizes the current weight group to be processed, it searches from the current search range when the weight group that has a greater impact on the accuracy of the model has been quantized: the target quantization result of the current weight group to be processed that minimizes the corresponding second model loss. That is, for the weight group that has a greater impact on the accuracy of the model, the electronic device will quantize the weight group that has a smaller impact on the accuracy of the model according to the quantization result of the weight group. Then the greater the impact of the weight group on the accuracy of the model, the more times the electronic device performs quantization based on the weight group, and each time the electronic device performs quantization based on the weight group, the quantization result is obtained under the condition of minimizing the model loss. In this way, the quantized target model can be obtained based on the quantization results of each weight group obtained, and the model loss of the target model is small.

[0194] The present application also provides an electronic device, such as Figure 6 As shown, it includes: a memory 601 for storing computer programs; a processor 602 for implementing the steps of any model quantization method in the above embodiments when executing the program stored in the memory 601.

[0195] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 602, the communication interface, and the memory 601 communicate with each other via the communication bus.

[0196] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0197] The communication interface is used for communication between the above electronic device and other devices.

[0198] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0199] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model quantization methods are implemented.

[0201] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any model quantization method in the above embodiments.

[0202] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0203] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0204] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0205] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model quantization method, characterized in that: The method comprises: For each network layer to be quantized in the original model, obtain a weight matrix of the network layer to be quantized in floating point format as the weight matrix to be quantized; Grouping the weights in the weight matrix to be quantized to obtain a plurality of weight groups; wherein the weights contained in a weight group correspond to the same input channel of the network layer to be quantized; For each weight group, using the target quantization coefficient of the weight group, according to the activation-perceptual weight quantization algorithm, calculate the model loss when the weight group is quantized to the target quantization format, as the first model loss corresponding to the weight group; wherein, among the candidate quantization coefficients, the model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group is the smallest; the target quantization format is determined based on the quantization format supported by the platform on which the original model is to be deployed; Determine the current weight group to be processed according to the order of the corresponding first model loss from large to small; Taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, the target quantization result of the current weight group to be processed is searched from the current search range determined based on the target quantization format; and returning to execute the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; wherein the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range; The target quantization results of each weight group are used to replace the weight matrix to be quantized of the network layer to be quantized in the original model to obtain a quantized target model.

2. The method according to claim 1, characterized in that The method of calculating, for each weight group, a model loss when quantizing the weight group to a target quantization format using a target quantization coefficient of the weight group according to an activation-aware weight quantization algorithm as a first model loss corresponding to the weight group includes: For each weight group, the average value of the absolute values ​​of the weights in the weight group is calculated to obtain the reference quantization coefficient of the weight group; Correcting the reference quantization coefficients based on the preset correction coefficients to obtain a plurality of candidate quantization coefficients; For each candidate quantization coefficient, according to the candidate quantization coefficient, the maximum value of the absolute values ​​of the weights in the weight group, and the target quantization format, calculate the candidate scaling coefficient corresponding to the candidate quantization coefficient; quantizing the weight group to the target quantization format according to the candidate scaling coefficient corresponding to the candidate quantization coefficient, and obtaining the candidate quantization result corresponding to the candidate quantization coefficient; Dequantizing the corresponding candidate quantization result using the candidate scaling coefficient corresponding to the candidate quantization coefficient to obtain a dequantization result; Calculate the model loss corresponding to the candidate quantization coefficient based on the difference between the obtained dequantization result and the weight group; The minimum model loss is determined from the calculated model losses as the first model loss when the weight group is quantized to the target quantization format according to the target quantization coefficient of the weight group.

3. The method according to claim 2, characterized in that Each preset correction coefficient is: the value of each average point in the data interval [0, 1]; The reference quantization coefficients are corrected based on the preset correction coefficients to obtain a plurality of candidate quantization coefficients, including: For each preset correction coefficient, the reference quantization coefficient is corrected according to a first correction formula using the preset correction coefficient to obtain a candidate quantization coefficient corresponding to the preset correction coefficient; Wherein, the first correction formula is: ； s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

4. The method according to claim 2, characterized in that: Each preset correction coefficient is: the value of each average point in the data interval [0, 1]; The reference quantization coefficients are corrected based on the preset correction coefficients to obtain a plurality of candidate quantization coefficients, including: For each preset correction coefficient, the reference quantization coefficient is corrected according to a second correction formula using the preset correction coefficient to obtain a candidate quantization coefficient corresponding to the preset correction coefficient; Wherein, the second correction formula is: ； s represents the candidate quantization coefficient corresponding to the preset correction coefficient; represents the reference quantization coefficient; Indicates the preset correction factor.

5. The method according to any one of claims 2 to 4, characterized in that: The step of calculating, for each candidate quantization coefficient, a candidate scaling coefficient corresponding to the candidate quantization coefficient according to the candidate quantization coefficient, the maximum value of the absolute values ​​of the weights in the weight group, and the target quantization format includes: For each candidate quantization coefficient, the candidate quantization coefficient, the maximum value of the absolute values ​​of the weights in the weight group, and the target quantization format are used to calculate the candidate scaling factor corresponding to the candidate quantization coefficient according to the scaling factor calculation formula; wherein the scaling factor calculation formula is: ； Indicates the candidate scaling coefficient corresponding to the candidate quantization coefficient; represents the product of the maximum absolute value of each weight in the weight group and the candidate quantization coefficient; N represents the number of bits indicated by the target quantization format.

6. The method according to claim 1, characterized in that The weights in the weight matrix to be quantized are grouped to obtain a plurality of weight groups, including: Determine, from the weight matrix to be quantized, the weights corresponding to the same input channel of the network layer to be quantized; The determined weights are divided into a plurality of groups to obtain a plurality of weight groups; wherein the weights included in a weight group are adjacent in the weight matrix to be quantized.

7. The method according to claim 1, characterized in that Before searching for a target quantization result of the current weight group to be processed within a current search range determined based on the target quantization format with the second model loss corresponding to the current weight group to be processed being minimized as the optimization goal, the method further includes: Obtaining an intermediate quantization result of the current weight group to be processed; wherein the intermediate quantization result is obtained by quantizing the current weight group to be processed to a target quantization format according to an activation-aware weight quantization algorithm using a target quantization coefficient of the current weight group to be processed; The step of taking the minimization of the second model loss corresponding to the weight group to be processed currently as the optimization goal and searching for the target quantization result of the weight group to be processed currently within the current search range determined based on the target quantization format includes: Within the data range indicated by the target quantization format, determining the data interval to which the intermediate quantization result of the weight group to be processed currently belongs as the current search range; Taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, the target quantization result of the current weight group to be processed is searched within the current search range.

8. The method according to claim 7, characterized in that The optimization objective is to minimize the second model loss corresponding to the current weight group to be processed, and to search for the target quantization result of the current weight group to be processed within the current search range, including: Taking the minimum loss of the second model corresponding to the current weight group to be processed as the optimization goal, the target quantization result of the current weight group to be processed is searched from the current search range in a gradient descent manner; or, Taking the minimization of the second model loss corresponding to the current weight group to be processed as the optimization goal, the target quantization result of the current weight group to be processed is searched in a traversal manner from the current search range.

9. A model quantization device, characterized in that: The device comprises: A matrix acquisition module is used to obtain a weight matrix of the network layer to be quantized in a floating point format for each network layer to be quantized in the original model as a weight matrix to be quantized; A grouping module, used for grouping the weights in the weight matrix to be quantized to obtain a plurality of weight groups; wherein the weights contained in a weight group correspond to the same input channel of the network layer to be quantized; A loss calculation module is used to calculate, for each weight group, a model loss when the weight group is quantized to a target quantization format using a target quantization coefficient of the weight group and according to an activation-perceptual weight quantization algorithm, as a first model loss corresponding to the weight group; wherein, among the candidate quantization coefficients, the model loss when the weight group is quantized to a target quantization format according to the target quantization coefficient of the weight group is the smallest; and the target quantization format is determined based on a quantization format supported by a platform on which the original model is to be deployed; A determination module, used to determine the current weight group to be processed according to the order of the corresponding first model loss from large to small; The target quantization result acquisition module is used to minimize the second model loss corresponding to the current weight group to be processed as the optimization target, and search for the target quantization result of the current weight group to be processed within the current search range determined based on the target quantization format; and return to execute the step of determining the current weight group to be processed in the order of the corresponding first model loss from large to small, until the target quantization result of each weight group is obtained; wherein the second model loss corresponding to the current weight group to be processed represents the loss between the current reference weight matrix and the weight matrix to be quantized; the current reference weight matrix is ​​obtained by replacing the corresponding weight group in the weight matrix to be quantized using the currently determined target quantization result and the quantization result of the current weight group to be processed found within the current search range; The replacement module is used to replace the weight matrix to be quantized of the network layer to be quantized in the original model by using the target quantization results of each weight group to obtain a quantized target model.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images