Model quantization method, apparatus, device, storage medium, and program product

Abstract

Embodiments of the present application disclose a model quantization method, device, equipment, storage medium and computer program product, wherein the method comprises: obtaining first output data of at least one first network substructure in a first network model after the first network model processes a calibration data set; obtaining second output data of at least one second network substructure in a second network model based on activation quantization identification of the second network substructure, by processing the calibration data set by using the second network model obtained after quantizing the first network model; adjusting parameters of the second network substructure based on the first output data of each first network substructure and the second output data of the second network substructure corresponding to the first network substructure; and determining the adjusted second network model as a third network model when it is determined that a preset condition is met.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of information technology, and in particular to a model quantization method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Modern deep learning techniques pursue higher performance by consuming more memory and computing power. While large models can be trained in the cloud, deploying models directly on edge devices is very difficult due to limited computing resources (including latency, energy, and memory consumption). Techniques such as model quantization, pruning, distillation, lightweight network design, and weight matrix factorization can accelerate the inference of deep models. Model quantization, in particular, converts the weights and activation values ​​in a neural network from their original floating-point values ​​to low-bit-width integers (e.g., 8-bit, 4-bit, 3-bit, 2-bit, etc.). The quantized neural network model requires less storage space, and the computational form changes from floating-point operations to the less costly computation of low-bit-width integer data.

[0003] In the model quantization schemes of related technologies, the quantized models have insufficient accuracy, especially when using low bit width (such as 3 bits, 2 bits, etc.) for model quantization, the accuracy of the model will drop significantly and cannot meet the application requirements. Summary of the Invention

[0004] In view of this, embodiments of this application provide a model quantization method, apparatus, device, storage medium, and computer program product.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] On the one hand, embodiments of this application provide a model quantization method, the method comprising:

[0007] Obtain first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model;

[0008] Using a second network model, the calibration dataset is processed based on the activation quantization flag of at least one second network substructure in the second network model to obtain the second output data of each second network substructure; wherein, the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0009] For each of the first network substructures, the parameters of the second network substructure are adjusted based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0010] If the preset conditions are met, the adjusted second network model will be determined as the third network model.

[0011] On the other hand, embodiments of this application provide a model quantization apparatus, the apparatus comprising:

[0012] The first acquisition module is used to acquire first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model;

[0013] The processing module is used to process the calibration dataset using a second network model and based on the activation quantization flag of at least one second network substructure in the second network model to obtain second output data for each second network substructure; wherein the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0014] The adjustment module is used to adjust the parameters of the second network substructure for each of the first network substructures, based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0015] The first determining module is used to determine the adjusted second network model as the third network model when it is determined that the preset conditions are met.

[0016] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0017] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0018] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the above-described method.

[0019] In another aspect, embodiments of this application provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method.

[0020] In this embodiment, first output data of at least one first network substructure in a first network model is obtained, where each first output data is obtained by processing a calibration dataset using the first network model. Using a second network model obtained after quantizing the first network model, the calibration dataset is processed based on the activation quantization flags of at least one second network substructure in the second network model to obtain second output data for each second network substructure. The activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized. For each first network substructure, the parameters of the second network substructure are adjusted based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model. If preset conditions are met, the adjusted second network model is determined as a third network model. In this way, during the adjustment of the parameters of each second network substructure in the quantized second network model, the activation quantization flag of each second network substructure determines whether to quantize its activation value. This allows the second network model to learn to reduce the impact of activation value quantization during the adjustment of each second network substructure. As a result, the accuracy of the quantized model can be improved, and the flatness of the model under quantization perturbation can be enhanced. This enables models quantized with different bit widths to achieve high accuracy, thereby meeting the model quantization requirements of different tasks and deployment devices, and improving the versatility of model quantization applications. Attached Figure Description

[0021] Figure 1 A schematic diagram illustrating the implementation process of a model quantization method provided in this application embodiment;

[0022] Figure 2 A schematic diagram illustrating the implementation process of a model quantization method provided in this application embodiment;

[0023] Figure 3 A schematic diagram illustrating the implementation process of a model quantization method provided in this application embodiment;

[0024] Figure 4 A schematic diagram illustrating the implementation process of a model quantization method provided in this application embodiment;

[0025] Figure 5AThis is a schematic diagram illustrating the implementation of adjusting the rounding method used for the quantization weight value in the k-th block structure, as provided in an embodiment of this application.

[0026] Figure 5B This is a schematic diagram illustrating the implementation of adjusting the rounding method used for the quantization weight value in the k-th block structure, as provided in an embodiment of this application.

[0027] Figure 5C This is a schematic diagram illustrating the implementation of adjusting the rounding method used for the quantization weight value in the k-th block structure, as provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the composition structure of a model quantization device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to be limiting of this application.

[0034] This application provides a model quantization method that can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 This is a schematic diagram illustrating the implementation process of a model quantization method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps S101 to S104:

[0035] Step S101: Obtain first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model.

[0036] Here, the first network model can be any suitable neural network model to be quantized, and can be a full-precision neural network model. For example, the first network model can be a neural network model with 32-bit floating-point parameters or 16-bit floating-point parameters. Of course, this application embodiment does not limit the number of floating-point bits of the first network model. In implementation, the first network model can adopt any suitable neural network structure, including but not limited to one or more of ResNet-18, ResNet-50, MobileNetV2, EfficientNet-Lite, RegNet, BERT, etc. The first network model can be implemented based on a convolutional neural network or based on a transformer network, and is not limited here.

[0037] The structure of a neural network model can include stages, blocks, and layers according to different granularities. Each neural network model can include at least one stage, each stage can include at least one block, and each block can include at least one layer. The processing layer can be, for example, an input layer, a convolutional layer, a pooling layer, a downsampling layer, a linear correction unit, a fully connected layer, a batch normalization layer, etc. In the embodiments of this application, the first network model can include at least one first network substructure. The first network substructure can be a single processing layer, a block structure including at least two processing layers, or a stage structure including at least two block structures. In implementation, those skilled in the art can determine the at least one first network substructure in the first network model using an appropriate granularity according to the actual situation; the embodiments of this application do not limit this.

[0038] The calibration dataset may include at least one image data, point cloud data, or voice data, etc. This application embodiment does not limit the type of data in the calibration dataset. In implementation, the calibration dataset may be pre-set or sampled from a specific dataset; this is not limited here.

[0039] By processing the calibration dataset using the first network model, output data (denoted as first output data) of at least one first network substructure in the first network model can be obtained. In implementation, the first output data of each first network substructure can be predetermined or obtained by processing the calibration dataset in real time; there is no limitation here.

[0040] Step S102: Using the second network model, based on the activation quantization identifier of at least one second network substructure in the second network model, the calibration dataset is processed to obtain the second output data of each second network substructure; wherein, the second network model is obtained by quantizing the first network model, and the activation quantization identifier of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0041] Here, the second network model is obtained by quantizing the first network model. In implementation, any suitable model quantization algorithm can be used to quantize the first network model to obtain the second network model. For example, the first network model can be pre-trained, and then a suitable quantization operation can be performed on the pre-trained first network model to obtain the second network model. Alternatively, the first network model can be quantized-awarely trained, and the resulting trained first network model is the quantized second network model. The second network model obtained by quantizing the first network model may have a significant quantization loss, which can be reduced through further optimization.

[0042] The structure of the second network model is the same as that of the first network model. That is, each first network substructure in the first network model has a corresponding second network substructure in the second network model that is identical to that first network substructure. The difference lies in that the parameters in the second network model are quantized to a preset bit width. For example, the second network model can be a quantized model of the first network model with a 1-bit or 2-bit width, a quantized model of the first network model with a 4-bit width, a quantized model of the first network model with an 8-bit width, and so on. Of course, the parameters of different second network substructures in the second network model can be quantized with different bit widths or with the same bit width; the quantization of weight values ​​and activation values ​​in the same second network substructure can use the same bit width or different bit widths, and this application embodiment does not limit this.

[0043] In processing the calibration dataset using the second network model, for each second network substructure in the second network model, an activation quantization flag can be used to characterize whether the activation value of the second network substructure is quantized. The activation quantization flag for each second network substructure can be a first flag indicating that the activation value of the second network substructure is quantized, or a second flag indicating that the activation value of the second network substructure is not quantized. During the processing of the calibration dataset using the second network model, the activation values ​​in the second network substructure with the first activation quantization flag can be quantized based on set quantization parameters, while the activation values ​​in the second network substructure with the second activation quantization flag are not quantized. In implementation, the activation quantization flag for each second network substructure in the second network model can be preset or dynamically determined during the processing of the calibration dataset using the second network model; this embodiment does not limit this. During multiple processing runs of the calibration dataset using the second network model, the activation quantization flags of each second network substructure in the second network model can be the same or different.

[0044] In some implementations, the activation quantization identifier of each second network substructure can be determined randomly during the processing of the calibration dataset using the second network model. For example, the activation quantization identifier of each second network substructure can be determined as a first identifier or a second identifier based on a set quantization probability.

[0045] In some implementations, the activation quantization identifier of each second network substructure can be pre-assigned according to specific quantization setting rules. For example, each second network substructure in the second network model can be sorted, and the activation quantization identifier of the second network substructure with an odd number can be determined as the first identifier, while the activation quantization identifier of the second network substructure with an even number can be determined as the second identifier.

[0046] By processing the calibration dataset using the second network model, the output data of at least one second network substructure in the second network model can be obtained (denoted as the second output data).

[0047] Step S103: For each of the first network substructures, adjust the parameters of the second network substructure based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0048] Here, for each first network substructure in the first network model, the parameters of the second network substructure can be adjusted at least once based on the output data of the first network substructure and the second output data of the corresponding second network substructure in the second network model. By adjusting the parameters of the second network substructure at least once, the loss between the second output data of each second network substructure and the first output data of the corresponding first network substructure can be reduced, thereby reducing the quantization loss of the second network model.

[0049] The adjustable parameters in the second network substructure may include, but are not limited to, at least one of the following: quantization parameters of each weight value in the second network substructure (such as quantization step size, preset precision of quantization scale, quantization symmetry, quantization bit width and quantization granularity, etc.), quantization parameters of each activation value, rounding method used to quantize each weight value in the second network substructure (such as rounding up or rounding down, etc.), and quantization function used to quantize the weight values ​​in the second network substructure.

[0050] In implementation, appropriate parameter optimization algorithms can be used to adjust the parameters of each second network substructure according to the actual situation, such as gradient descent method, simulated annealing method, etc., but the embodiments of this application are not limited to this.

[0051] Step S104: If the preset conditions are met, the adjusted second network model is determined as the third network model.

[0052] Here, the preset conditions may include, but are not limited to, at least one of the following: the number of times the parameters of each second network substructure are adjusted reaches a set threshold; the loss value between each first output data and the corresponding second output data is less than a set first loss threshold; the total loss value between each first output data and the corresponding second output data is less than a set second loss threshold.

[0053] In some implementations, the parameters of each second network substructure in the second network model can be adjusted at least once. If the adjustment of the parameters of each second network substructure in the second network model meets the preset conditions, the adjusted second network model is determined as the third network model.

[0054] In some implementations, if the preset condition is not met, the calibration dataset can be processed using the adjusted second network model based on the activation quantization flag of at least one second network substructure in the second network model to obtain new second output data for each second network substructure; and for each first network substructure, the parameters of the second network substructure can be readjusted based on the first output data of the first network substructure and the new second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0055] In this embodiment, first output data of at least one first network substructure in a first network model is obtained, where each first output data is obtained by processing a calibration dataset using the first network model. Using a second network model obtained after quantizing the first network model, the calibration dataset is processed based on the activation quantization flags of at least one second network substructure in the second network model to obtain second output data for each second network substructure. The activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized. For each first network substructure, the parameters of the second network substructure are adjusted based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model. If preset conditions are met, the adjusted second network model is determined as a third network model. In this way, during the adjustment of the parameters of each second network substructure in the quantized second network model, the activation quantization flag of each second network substructure determines whether to quantize its activation value. This allows the second network model to learn to reduce the impact of activation value quantization during the adjustment of each second network substructure. As a result, the accuracy of the quantized model can be improved, and the flatness of the model under quantization perturbation can be enhanced. This enables models quantized with different bit widths to achieve high accuracy, thereby meeting the model quantization requirements of different tasks and deployment devices, and improving the versatility of model quantization applications.

[0056] In some implementations, each of the second network substructures is a processing layer. In this way, the parameters in the second network model can be adjusted according to the granularity of the processing layer. This allows for learning at the granularity of the processing layer to reduce the impact of activation value quantization, thereby improving the accuracy of parameter adjustment in the second network model and further enhancing the accuracy of the quantized model.

[0057] In some embodiments, determining that the preset condition is met in step S104 above may include at least one of the following steps S111 and S112:

[0058] Step S111: If, based on each of the first output data and each of the second output data, the loss value of each of the second network substructures is determined to satisfy a preset loss constraint, then the preset condition is satisfied.

[0059] Here, the loss value of each second network substructure can be the loss value between the second output data of that second network substructure and the first output data of the corresponding first network substructure in the adjusted second network model. The loss constraint can be a pre-defined target for adjusting the second network model, and can include constraints on the loss values ​​of each second network substructure in the adjusted second network model. For example, the loss constraint can include, but is not limited to, at least one of the following: the loss value of each second network substructure is less than a set first loss threshold, or the sum of the loss values ​​of all second network substructures in the second network model is less than a set second loss threshold.

[0060] Step S112: If the number of times each of the second network substructures is adjusted meets the preset number constraint, then the preset condition is determined to be satisfied.

[0061] Here, the frequency constraint can be a pre-defined target for adjusting the second network model, which may include a constraint on the number of times each second network substructure is adjusted. For example, the frequency constraint may include, but is not limited to, at least one of the following: the number of adjustments to each second network substructure all reach a set first frequency threshold; the maximum value of the number of adjustments to each second network substructure reaches a set second frequency threshold; the average number of adjustments to each second network substructure reaches a set third frequency threshold.

[0062] In implementation, those skilled in the art can set appropriate loss constraints or number constraints according to the actual situation, and the embodiments of this application are not limited in this regard.

[0063] In some embodiments, the above method may further include:

[0064] Step S121: Quantize each first network substructure in the first network model according to at least one bit width to obtain a second network model; wherein, the second network model includes at least one second network substructure corresponding to each first network substructure, and each second network substructure includes one of the following: stage structure, block structure, and processing layer.

[0065] This allows for adjustments to the parameters in the second network model at different granularities, enabling learning at different granularities to reduce the impact of activation value quantization and thus improving the flexibility of parameter adjustments in the second network model.

[0066] In some implementations, the at least one bit width may include, but is not limited to, at least one of 1-bit, 2-bit, 3-bit, 4-bit, and 8-bit widths.

[0067] In the above embodiments, it is possible to adjust the parameters of the second network substructure in a second network model quantized with a bit width of 1 bit, 2 bit, 3 bit, 4 bit, and / or 8 bit, so that the quantized model can achieve high accuracy. This can meet the model quantization requirements of various tasks and deployment devices, and improve the versatility of model quantization applications. For example, it is possible to adjust the parameters of the second network substructure in a second network model quantized with extremely low bit widths such as 1 bit, 2 bit, and 3 bit, so that the accuracy of the model after extremely low bit width quantization is effectively improved. This can improve the model accuracy while reducing the storage and computing resources required to deploy the quantized model.

[0068] In some embodiments, the above method may further include the following steps S131 to S133:

[0069] Step S131: Sample at least one candidate sample from the set candidate dataset.

[0070] Here, the candidate dataset can be pre-defined, including at least one sample used to calibrate the second network model. In implementation, any suitable sampling method can be used to sample at least one candidate sample from the candidate dataset; this application embodiment is not limited in this regard. For example, a set number of candidate samples can be randomly sampled from the candidate dataset, at least one candidate sample can be uniformly sampled from the candidate dataset, or at least one candidate sample can be selected from the candidate dataset according to set filtering conditions.

[0071] Step S132: Perform data augmentation processing on each candidate sample to obtain at least one target sample.

[0072] Here, for each candidate sample, any suitable data augmentation process can be applied to obtain the corresponding target sample. For example, data augmentation processes may include, but are not limited to, at least one of random flipping, random pruning, and adding perturbations.

[0073] In practice, the same data augmentation process can be applied to different candidate samples, or different data augmentation processes can be applied to different candidate samples. This application does not limit this.

[0074] Step S133: Based on the at least one target sample, obtain the calibration dataset.

[0075] Here, some or all of the target samples can be added to the calibration dataset.

[0076] In the above embodiments, at least one candidate sample is sampled from a set candidate dataset, and data augmentation is performed on each candidate sample to obtain at least one target sample. Based on at least one target sample, a calibration dataset is obtained. In this way, since the data in the calibration dataset is obtained after data augmentation of the candidate samples, the diversity of the samples in the calibration dataset can be increased, thereby further enhancing the ability of the second network model to learn to reduce the impact of activation value quantization, and thus further improving the flatness of the model under quantization perturbation.

[0077] This application provides a model quantization method, which can be executed by a computer device's processor. For example... Figure 2 As shown, the method includes the following steps S201 to S205:

[0078] Step S201: Obtain first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model.

[0079] Here, step S201 corresponds to step S101 mentioned above, and the specific implementation of step S101 can be referred to during implementation.

[0080] Step S202: The calibration dataset is determined as the input data for the first second network substructure in the second network model.

[0081] Here, the calibration dataset can be input into the second network model, which is the input data for the first second network substructure in the second network model.

[0082] Step S203: For each second network substructure in the second network model, the input data of the second network substructure is processed based on the weight value and activation quantization flag of the second network substructure to obtain the second output data of the second network substructure, and the second output data is used as the input data of the next second network substructure.

[0083] Here, the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure represents whether the activation value of the second network substructure is quantized.

[0084] In practice, the second output data of each second network substructure in the second network model can be used as the input data of the next second network substructure.

[0085] By inputting the input data of each second network substructure into the second network substructure, the second output data of the second network substructure can be obtained by processing the input data based on the weight value and activation quantization flag of the second network substructure.

[0086] In some implementations, each second network substructure includes at least one processing layer. The input data of each second network substructure can be used as the input data of the first processing layer within that second network substructure, and the output data of each processing layer can be used as the input data of the next processing layer. The output data of the last processing layer in the second network substructure is the second output data of that second network substructure. For each processing layer in the second network substructure, the input data of that processing layer can be processed based on its weight value to obtain the activation value of that processing layer. If the activation quantization flag of the second network substructure is a first flag indicating that the activation value of the second network substructure is quantized, the activation value of the processing layer can be quantized based on a set quantization parameter, and the quantized activation value is determined as the output data of that processing layer. If the activation quantization flag of the second network substructure is a second flag indicating that the activation value of the second network substructure is not quantized, the activation value can be determined as the output data of that processing layer.

[0087] Step S204: For each of the first network substructures, adjust the parameters of the second network substructure based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0088] Step S205: If the preset conditions are met, the adjusted second network model is determined as the third network model.

[0089] Here, steps S204 to S205 correspond to steps S103 to S104, respectively. In practice, the specific implementation of steps S103 to S104 can be referred to.

[0090] In this embodiment, the calibration dataset is determined as the input data for the first second network substructure in the second network model. For each second network substructure in the second network model, the input data of that second network substructure is processed based on its weight values ​​and activation quantization flags to obtain its second output data. This second output data is then used as the input data for the next second network substructure. This allows for the rapid and accurate acquisition of the second output data for each second network substructure.

[0091] In some embodiments, the process of processing the input data of the second network substructure based on the weight values ​​and activation quantization flags of the second network substructure to obtain the second output data of the second network substructure in step S203 above may include the following steps S211 to S212:

[0092] Step S211: Based on the rounding method of the weight values ​​in the second network substructure and the first quantization parameter, the weight values ​​of the second network substructure are quantized to obtain the quantized weight values.

[0093] Here, the rounding method for the weight value can include, but is not limited to, rounding up, rounding down, or rounding to the nearest whole number.

[0094] The first quantization parameter can be a quantization parameter used to quantize the weight values ​​in the second network substructure, including but not limited to at least one of quantization step size, preset precision of quantization scale, quantization symmetry, quantization bit width, and quantization granularity. In some embodiments, the calibration dataset can be pre-processed using the second network model to obtain the output data of the second network model, and the first quantization parameter used to quantize the weight values ​​in the second network substructure can be obtained by statistically analyzing the output data. In some embodiments, the first quantization parameter used to quantize the weight values ​​in the second network substructure can be an adjustable parameter of the second network substructure, and the second quantization parameter can be learned during the adjustment of the parameters of the second network substructure.

[0095] Step S212: Based on the quantized weight values ​​and the activation quantization flag of the second network substructure, the input data of the second network substructure is processed to obtain the second output data of the second network substructure.

[0096] In some implementations, for each processing layer in the second network substructure, the input data of the processing layer can be processed based on the quantized weight value of the processing layer to obtain the activation value of the processing layer. When the activation quantization identifier of the second network substructure is a first identifier indicating that the activation value of the second network substructure is quantized, the activation value of the processing layer can be quantized based on the set quantization parameters, and the quantized activation value can be determined as the output data of the processing layer. When the activation quantization identifier of the second network substructure is a second identifier indicating that the activation value of the second network substructure is not quantized, the activation value can be determined as the output data of the processing layer.

[0097] In the above embodiments, the weight values ​​in the second network substructure can be quantized by rounding the weight values ​​in each second network substructure and using the first quantization parameter. Based on the activation quantization flag of each second network substructure, it can be determined whether to quantize the activation values ​​in that second network substructure. On this basis, the input data of each second network substructure can be processed to obtain the second output data of each second network substructure.

[0098] In some embodiments, when the activation quantization identifier of the second network substructure is a first identifier, during the processing of the input data of the second network substructure, the activation values ​​in the second network substructure are quantized based on a second quantization parameter. Here, the second quantization parameter can be a quantization parameter used to quantize the activation values ​​in the second network substructure, including but not limited to at least one of quantization step size, preset precision of the quantization scale, quantization symmetry, quantization bit width, and quantization granularity. In some embodiments, the calibration dataset can be pre-processed using the second network model to obtain the output data of the second network model, and the second quantization parameter used to quantize the activation values ​​in the second network substructure can be obtained by statistically analyzing the output data. In some embodiments, the second quantization parameter used to quantize the activation values ​​in the second network substructure can be an adjustable parameter of the second network substructure, and the second quantization parameter can be learned during the adjustment of the parameters of the second network substructure.

[0099] In some embodiments, when the activation quantization identifier of the second network substructure is a second identifier, the activation values ​​in the second network substructure are not quantized during the processing of the input data of the second network substructure. Thus, by setting the activation quantization identifier of at least one second network substructure in the second network model to the second identifier, the activation values ​​of that at least one second network substructure are not quantized during the processing of calibration data using the second network model. This increases the diversity of the impact of activation value quantization on each second network substructure, thereby further enhancing the ability of the second network model to learn to reduce the impact of activation value quantization. This further improves the flatness of the model under quantization perturbations, resulting in a more universally flat second network model, ultimately improving model accuracy.

[0100] This application provides a model quantization method, which can be executed by a computer device's processor. For example... Figure 3 As shown, the method includes the following steps S301 to S305:

[0101] Step S301: Obtain first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model.

[0102] Here, step S301 corresponds to step S101 mentioned above, and the specific implementation of step S101 mentioned above can be referred to during implementation.

[0103] Step S302: For each second network substructure in the second network model, based on the set probability distribution parameters, the activation quantization flag of the second network substructure is randomly assigned a value.

[0104] Here, the probability distribution parameter can be any suitable parameter that can characterize the probability of at least one value of the activation quantization identifier of the second network substructure. This can include, but is not limited to, at least one of the following: the probability of assigning the activation quantization identifier of the second network substructure to the first identifier, the probability of assigning the activation quantization identifier of the second network substructure to the second identifier, etc. Based on this probability distribution parameter, the activation quantization identifier of each second network substructure can be randomly assigned to either the first identifier or the second identifier, etc.

[0105] In implementation, the probability distribution parameters can be preset. Those skilled in the art can set appropriate probability distribution parameters according to the actual situation. This application does not limit this.

[0106] Step S303: Using the second network model, based on the activation quantization identifier of at least one second network substructure in the second network model, the calibration dataset is processed to obtain the second output data of each second network substructure; wherein, the second network model is obtained by quantizing the first network model, and the activation quantization identifier of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0107] Step S304: For each of the first network substructures, adjust the parameters of the second network substructure based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0108] Step S305: If the preset conditions are met, the adjusted second network model is determined as the third network model.

[0109] Here, steps S303 to S305 correspond to steps S102 to S104, respectively. In practice, the specific implementation of steps S102 to S104 can be referred to.

[0110] In some embodiments, the random assignment of the activation quantization flag of the second network substructure based on the set probability distribution parameters in step S302 above may include at least one of steps S311 and S312:

[0111] Step S311: When the probability distribution parameter includes the quantization probability, the activation quantization identifier of the second network substructure is randomly assigned a value based on the quantization probability; wherein, the quantization probability represents the probability that the activation quantization identifier of the second network substructure is assigned a first identifier, and the first identifier represents the quantization of the activation value of the corresponding second network substructure.

[0112] Here, according to the quantization probability, the activation quantization flag of the second network substructure can be randomly assigned to a first flag or a second flag, etc. In implementation, those skilled in the art can pre-set an appropriate quantization probability according to the actual situation; for example, the quantization probability can be 0, 0.25, 0.5, 0.75, or 1, etc.

[0113] For example, when the quantization probability is p, the activation quantization identifier of the second network substructure can be randomly assigned to either the first identifier or the second identifier based on the quantization probability p. The probability of assigning the activation quantization identifier of the second network substructure to the first identifier is the quantization probability p, and the probability of assigning the activation quantization identifier of the second network substructure to the second identifier is 1-p.

[0114] Step S312: When the probability distribution parameter includes the quantized inactivation probability, the activation quantization identifier of the second network substructure is randomly assigned a value based on the quantized inactivation probability; wherein, the quantized inactivation probability represents the probability that the activation quantization identifier of the second network substructure is assigned a second identifier, and the second identifier represents that the activation value of the corresponding second network substructure is not quantized.

[0115] Here, the activation quantization flag of the second network substructure can be randomly assigned to either the first flag or the second flag according to the quantization inactivation probability. In implementation, those skilled in the art can pre-set an appropriate quantization inactivation probability based on the actual situation. For example, the quantization inactivation probability can be 0, 0.25, 0.5, 0.75, or 1, etc.

[0116] For example, when the quantization inactivation probability is q, the activation quantization identifier of the second network substructure can be randomly assigned to either the first identifier or the second identifier based on the quantization inactivation probability q. The probability of assigning the activation quantization identifier of the second network substructure to the first identifier is 1-q, and the probability of assigning the activation quantization identifier of the second network substructure to the second identifier is the quantization inactivation probability q.

[0117] It should be noted that the same quantization probability and / or quantization inactivation probability can be set for each second network substructure in the second network model, or different quantization probabilities and / or quantization inactivation probabilities can be set for each second network substructure in the second network model. This application embodiment does not limit this.

[0118] In the above embodiments, based on the set probability distribution parameters, the activation quantization identifier of each second network substructure in the second network model is randomly assigned a value, such that the activation quantization identifier of each second network substructure is randomly assigned a first identifier or a second identifier according to the probability distribution parameters. This increases the randomness of quantizing the activation values ​​in each second network substructure, thus increasing the diversity of the impact of activation value quantization on quantization loss. This further enhances the second network model's ability to learn to reduce the impact of activation value quantization, thereby further improving the model's flatness under quantization perturbations and leading to a further improvement in model accuracy.

[0119] This application provides a model quantization method, which can be executed by a computer device's processor. For example... Figure 4 As shown, the method includes the following steps S401 to S404:

[0120] Step S401: Obtain first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model.

[0121] Step S402: Using the second network model, based on the activation quantization identifier of at least one second network substructure in the second network model, the calibration dataset is processed to obtain the second output data of each second network substructure; wherein, the second network model is obtained by quantizing the first network model, and the activation quantization identifier of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0122] Here, steps S401 to S402 correspond to steps S101 to S102 mentioned above, and specific implementation methods of steps S101 to S102 can be referred to during implementation.

[0123] Step S403: For each of the first network substructures, based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model, determine the loss value of the second network substructure, and adjust the parameters of the second network substructure based on the loss value.

[0124] Here, the loss value of each second network substructure can be the loss value between the second output data of that second network substructure and the first output data of the corresponding first network substructure in the adjusted second network model. By comparing the second output data of the second network substructure with the first output data of the corresponding first network substructure, the loss value of that second network substructure can be obtained.

[0125] In some implementations, the calibration dataset includes at least one target sample, the first output data may include first sub-data obtained by processing each target sample, and the second output data may include second sub-data obtained by processing each target sample. For each target sample, the loss value of the second network substructure for processing the target sample can be determined based on the similarity between the first sub-data and the second sub-data corresponding to the target sample. The loss value of the second network substructure can be determined based on the mean square error between the loss values ​​of the second network substructure for processing each target sample.

[0126] In some implementations, the parameters of the second network substructure can be adjusted if the loss value of the second network substructure is less than a set third loss threshold.

[0127] In some implementations, the parameters of the second network substructure can be adjusted once it is determined that the change in the loss value of the second network substructure has converged.

[0128] Step S404: If the preset conditions are met, the adjusted second network model is determined as the third network model.

[0129] Here, step S404 corresponds to step S104 mentioned above, and the specific implementation of step S104 can be referred to during implementation.

[0130] In this embodiment, for each first network substructure, based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model, the loss value of the second network substructure is determined, and based on the loss value, the parameters of the second network substructure are adjusted. In this way, the consistency between the output of the second network substructure and the corresponding first network substructure can be maintained by adjusting the parameters of the second network substructure, thereby effectively improving the accuracy of the quantized model.

[0131] In some embodiments, the parameters of the second network substructure include the rounding method used to quantize the weight values ​​in the second network substructure; the adjustment of the parameters of the second network substructure based on the loss value in step S403 above includes:

[0132] Step S411: Based on the loss value, update the rounding method used to quantize the weight values ​​in the second network substructure, so that during the processing of the calibration dataset, the weight values ​​in the second network substructure are quantized using a quantization function corresponding to the rounding method.

[0133] Here, based on the rounding method used to quantize the weight values ​​in the second network substructure, the corresponding quantization function can be determined. During the processing of the calibration dataset, this quantization function can be used to quantize the weight values ​​in the second network substructure to a specific bit width. In implementation, a suitable optimization algorithm can be used to update the rounding method used to quantize the weight values ​​in the second network substructure, such as gradient descent or simulated annealing, depending on the actual situation. This application embodiment is not limited in this regard.

[0134] In the above embodiments, the rounding method used to quantize the weight values ​​in the second network substructure is updated based on the loss value of the second network substructure. Thus, by updating the rounding method for weight quantization, fine-tuning of the quantized second network model can be achieved, thereby further improving the accuracy of the adjusted second network model (i.e., the third network model).

[0135] Model quantization methods in related technologies are mainly divided into two categories: quantization-aware training and post-training quantization. Quantization-aware training requires a complete model training process, necessitating a large amount of sample data and significant GPU computing power for end-to-end model training. Post-training quantization, on the other hand, can obtain a quantized model based on a small amount of sample data and limited GPU computing power. Post-training quantization schemes in related technologies can round down the weight values ​​in a pre-trained, full-precision network model. By modeling the quantization of weight values ​​as noise and constructing a quantization optimization objective, the scheme learns the rounding method (such as rounding up or down) used for quantizing weight values. However, post-training quantization schemes in related technologies do not perform well for quantization with low bit widths such as 2-bit and 3-bit, and for detection tasks that are more complex than classification tasks, resulting in quantized models with insufficient accuracy.

[0136] In implementing this application, the inventors discovered through research that the post-training quantization schemes in related technologies only model the quantization of weight values ​​as noise and fine-tune the quantized weight values ​​by adjusting the rounding method used for quantization and reconstructing the output of each block structure / processing layer (i.e., weight fine-tuning). The quantization parameters (such as step size) for quantizing activation values ​​are determined after weight fine-tuning, and the quantization of activation values ​​is not considered during weight fine-tuning. In other words, weight fine-tuning and activation value quantization in related technologies are independent. For example, the weight values ​​obtained in the quantized model are the same whether the activation values ​​of a full-precision network model are quantized to a 2-bit width or a 3-bit width.

[0137] In view of this, embodiments of this application provide a model quantization method. This method introduces the quantization of activation values ​​during weight fine-tuning, which can effectively improve the performance of the quantized model and make the quantized model flatter in some directions (that is, the quantization loss of the quantized model obtained under the quantization perturbation of weights changes relatively little). In some embodiments, the quantization of activation values ​​can be carried out during the random deactivation weight fine-tuning, that is, randomly determining whether to quantize the activation values ​​in the model to be adjusted (corresponding to the second network model in the aforementioned embodiments) during the weight fine-tuning period. This can improve the diversity of the impact of activation value quantization on each second network substructure, so that the quantized and adjusted model (corresponding to the third network model in the aforementioned embodiments) has a flatness from a general perspective, thereby improving accuracy.

[0138] The following sections describe the impact of activation value quantization on weight fine-tuning during the weight fine-tuning period in the model quantization method provided in this application, as well as the method of quantizing activation value during weight fine-tuning.

[0139] 1) The impact of activation value quantization on weight fine-tuning:

[0140] To investigate the impact of activation value quantization on weight fine-tuning during the process, the inventors conducted preliminary experiments on the ImageNet dataset. In these experiments, at least one processing layer of the full-precision network model to be quantized (corresponding to the first network model) was treated as a block structure (e.g., the bottleneck layer was treated as a block structure), and the rounding method used for the quantized weight values ​​in each block structure was adjusted to adjust the quantized weights.

[0141] Taking the first network model, which includes K block structures, as an example, before adjusting the rounding method used to quantize the weight values ​​in the k-th block structure, we consider whether the activation values ​​from the 1st block structure to the k-th block structure are quantized separately during the model inference process, and compare the accuracy of the quantized and adjusted models in the following three schemes:

[0142] Option 1: As Figure 5A As shown, before adjusting the rounding method used for the quantization weight values ​​in the k-th block structure, the activation values ​​in the 1st to kth block structures are not quantized.

[0143] Option 2: As Figure 5B As shown, before adjusting the rounding method used for the quantization weight values ​​in the k-th block structure, the activation values ​​in the 1st to kth block structures are quantized.

[0144] Option 3: such as Figure 5C As shown, before adjusting the rounding method used for the quantization weight value in the k-th block structure, the activation values ​​in the 1st to (k-1)th block structures are quantized, but the activation value in the k-th block structure is not quantized.

[0145] Where K is a positive integer greater than 1, and k is greater than 1 but does not exceed K.

[0146] In the experiment, the second network model was obtained by quantizing the first network model. The rounding method used for the weight values ​​in each quantized block of the second network model was adjusted using schemes 1 to 3, respectively. The performance of the corresponding third network models is shown in Table 1. For the first network model using ResNet-18, the accuracies of the third network models obtained based on schemes 1, 2, and 3 are 18.88, 45.74, and 48.07, respectively. For the first network model using ResNet-50, the accuracies of the third network models obtained based on schemes 1, 2, and 3 are 4.3, respectively. 4. The accuracies of the first network model using MobileNetV2, and the third network models obtained based on schemes 1, 2, and 3, are 5.83, 50.71, and 51.20, respectively. For the first network model using RegNet-600MF, the accuracies of the third network models obtained based on schemes 1, 2, and 3 are 42.77, 60.94, and 62.07, respectively. For the first network model using MnasNet, the accuracies of the third network models obtained based on schemes 1, 2, and 3 are 26.62, 58.79, and 60.19, respectively. For the first network models using ResNet-18 and ResNet-50, the calculations are based on quantization of both weights and activation values ​​using a 2-bit width. For the first network models using RegNet-600MF, MobileNetV2, and MnasNet, the calculations are based on quantization of both weights and activation values ​​using a 3-bit width.

[0147] Table 1 shows examples of the accuracy of the third network models obtained using schemes 1 to 3 respectively.

[0148] Neural network structure Option 1 Option 2 Option 3 ResNet-18 18.88 45.74 48.07 ResNet-50 4.34 46.98 49.07 MobileNetV2 5.83 50.71 51.20 RegNet-600MF 42.77 60.94 62.07 MasNet 26.62 58.79 60.19

[0149] In the above experiments, for extremely low-bit quantization (e.g., quantizing both weights and activations with a width of 2 or 3 bits), the accuracy of the third network models obtained using schemes 2 and 3 significantly exceeded that of the third network model obtained using scheme 1. This demonstrates that considering activation quantization during weight fine-tuning allows the weight fine-tuning to learn to reduce the impact of activation quantization, resulting in a substantial performance improvement. Furthermore, the accuracy of the third network model obtained using scheme 3 is higher than that obtained using scheme 2, indicating that partially introducing activation quantization during weight fine-tuning is more effective than fully introducing it.

[0150] To further explore how activation quantization affects weight fine-tuning, the final loss target can be analyzed by simultaneously simulating the quantization of weight values ​​and activation values ​​as noise.

[0151] Typically, quantization noise can be... To indicate, among which, Let be the quantized parameter and 'a' be the full-precision parameter. However, the noise range represented in this way is affected by the parameter range or the quantization step size. To eliminate this effect, the activation quantization noise u, which is simulated by quantizing the activation value, can be compared with the quantized activation value. The relationship between the full-precision activation values ​​'a' is expressed in multiplicative form: This is formally equivalent to addition. Thus, the optimization objective of model quantization can be represented by the following formula 1:

[0152]

[0153] Where x represents the input data of the block structure in the second network model to be fine-tuned, which is derived from the calibration dataset D. c The value obtained from sampling is u(x), which represents the activation quantization noise introduced by quantizing the activation value when the input data is x, and w is the full-precision weight value in this block structure. Let L(w,x,1) be the quantized weight value in this block structure, and let L(w,x,1) be the output data of this block structure in the full-precision network model corresponding to the second network model. This refers to the output data in the second network model after quantizing the activation values ​​(i.e., after introducing activation quantization noise). Let represent the mean squared error function. The optimization objective is to optimize the rounding method used to quantize the weight values ​​of each block in the second network model, in order to find a . Make Minimum.

[0154] Compared to the scheme that does not consider activation value quantization in the weight fine-tuning stage, this optimization objective adds a term related to activation value quantization. For ease of analysis, the activation quantization noise u(x) is transferred to the weight quantization noise, and the transferred weight quantization noise can be expressed as 1+v(x). It can be converted into the form shown in Formula 2 below:

[0155]

[0156] Further transformation of Formula 2 above can yield the following: Convert to the form shown in Formula 3 below:

[0157]

[0158] In this context, term (3-1) represents the optimization objective in the weight fine-tuning stage without considering activation value quantization, while term (3-2) introduces a new term after quantizing activation values. Optimizing term (3-2) improves the flatness of the quantized weight values, meaning the change in network loss caused by the perturbation 1+v(x) is relatively small. In other words, for the input data x, for the activation quantization noise u(x), there exists a corresponding weight quantization noise v(x), making the quantized model flatter under the perturbation of the weight quantization noise v(x). For example, both Schemes 2 and 3 introduce activation value quantization in the weight fine-tuning stage, which allows for additional optimization of term (3-2) in Formula 3, resulting in a flatter quantized model.

[0159] Regarding the optimization objective of model quantization expressed in Equation 3 above, post-training quantization (i.e., offline quantization) is highly sensitive to calibration data. In practical applications, it is necessary not only to obtain a flatter quantized model on the calibration dataset but also to minimize the loss of the optimization objective on the test dataset. In implementation, the direction of model flatness can be diversified by varying the weights used to quantize the noise v(x) to meet the required model flatness on the test dataset, thereby ensuring that the quantized model performs well on the test dataset.

[0160] Inspired by the experimental results of schemes 1 to 3, since scheme 3 does not quantize activation values ​​in some block structures (i.e., sets some activation quantization noise u(x) to 0), it slightly alters the weight quantization noise v(x), making the flatness of the quantized model not limited to the calibration dataset. Therefore, in some implementations, the weight quantization noise v(x) can be modified by randomly discarding some activation quantization noise u(x), thereby introducing different weight quantization noise v(x) during the weight fine-tuning stage. Especially for each forward propagation, by randomly discarding some activation quantization noise u(x) (i.e., randomly determining whether to quantize the activation values ​​of each block structure during weight fine-tuning), the diversity of weight quantization noise v(x) can be effectively improved, thereby further improving the performance of the quantized model on the test dataset and giving the quantized model flatness from a general perspective. It can also further extend offline quantization to 2-bit wide quantization.

[0161] The model quantization method provided in this application, compared with the model quantization methods in related technologies, can achieve a 3% accuracy improvement on the ImageNet classification task using a 4-bit quantization width. In particular, for some lightweight network models more suitable for mobile devices, an accuracy improvement of up to 51.49% can be achieved using a 2-bit quantization width. On the MS COCO dataset, for the detection task, the accuracy of the two-stage Faster R-CNN model quantized with a 4-bit quantization width is similar to that of the full-precision floating-point model. For the single-stage RetinaNet, with a weight quantization width of 2 bits and an activation quantization width of 4 bits, an increase of up to 6.5 in mean average precision (mAP) can be achieved.

[0162] The model quantization method provided in this application can be applied to various scenarios, such as large model deployment, model application on edge devices, classification tasks, and detection tasks, and is applicable to different tasks, different models, and different quantization bit widths. Model quantization based on the method provided in this application can, on the one hand, ensure that the quantized model meets the requirements for inference speed while maintaining high accuracy; on the other hand, it can ensure that the quantized model meets the requirements for low power consumption.

[0163] Figure 6 This is a schematic diagram of the composition structure of a model quantization device provided in an embodiment of this application, as shown below. Figure 6 As shown, the model quantization device 600 includes: a first acquisition module 610, a processing module 620, an adjustment module 630, and a first determination module 640, wherein:

[0164] The first acquisition module 610 is used to acquire first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing the calibration dataset using the first network model;

[0165] The processing module 620 is used to process the calibration dataset using a second network model and based on the activation quantization flag of at least one second network substructure in the second network model to obtain second output data for each second network substructure; wherein the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized.

[0166] The adjustment module 630 is used to adjust the parameters of the second network substructure for each of the first network substructures based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model.

[0167] The first determining module 640 is used to determine the adjusted second network model as the third network model when it is determined that the preset conditions are met.

[0168] In some embodiments, the processing module is further configured to: determine the calibration dataset as input data for the first second network substructure in the second network model; for each second network substructure in the second network model, process the input data of the second network substructure using the second network substructure, based on the weight value and activation quantization flag of the second network substructure, to obtain the second output data of the second network substructure, and use the second output data as input data for the next second network substructure.

[0169] In some embodiments, the processing module is further configured to: quantize the weight values ​​of the second network substructure based on the rounding method of the weight values ​​in the second network substructure and the first quantization parameter to obtain quantized weight values; and process the input data of the second network substructure based on the quantized weight values ​​and the activation quantization flag of the second network substructure to obtain the second output data of the second network substructure.

[0170] In some embodiments, when the activation quantization identifier of the second network substructure is a first identifier, the activation value in the second network substructure is quantized based on a second quantization parameter during the processing of the input data of the second network substructure; and / or, when the activation quantization identifier of the second network substructure is a second identifier, the activation value in the second network substructure is not quantized during the processing of the input data of the second network substructure.

[0171] In some embodiments, the apparatus further includes: a second acquisition module, configured to randomly assign a value to the activation quantization identifier of the second network substructure based on a set probability distribution parameter for each second network substructure in the second network model.

[0172] In some embodiments, the second acquisition module is further configured to: when the probability distribution parameter includes a quantization probability, randomly assign a value to the activation quantization identifier of the second network substructure based on the quantization probability, wherein the quantization probability represents the probability that the activation quantization identifier of the second network substructure is assigned a first identifier, and the first identifier represents quantization of the activation value of the corresponding second network substructure; and / or, when the probability distribution parameter includes a quantization inactivation probability, randomly assign a value to the activation quantization identifier of the second network substructure based on the quantization inactivation probability; wherein the quantization inactivation probability represents the probability that the activation quantization identifier of the second network substructure is assigned a second identifier, and the second identifier represents that the activation value of the corresponding second network substructure is not quantized.

[0173] In some embodiments, the adjustment module is further configured to: determine the loss value of the second network substructure based on the first output data and the second output data; and adjust the parameters of the second network substructure based on the loss value.

[0174] In some embodiments, the parameters of the second network substructure include the rounding method used to quantize the weight values ​​in the second network substructure; the adjustment module is further configured to: update the rounding method used to quantize the weight values ​​in the second network substructure based on the loss value, so as to quantize the weight values ​​in the second network substructure using a quantization function corresponding to the rounding method during the processing of the calibration dataset.

[0175] In some embodiments, the first determining module is further configured to: determine that the preset condition is satisfied when the loss value of each second network substructure satisfies a preset loss constraint based on each first output data and each second output data; and determine that the preset condition is satisfied when the number of times each second network substructure is adjusted satisfies a preset number constraint.

[0176] In some embodiments, the apparatus further includes: a quantization module, configured to quantize each first network substructure in the first network model according to at least one bit width to obtain a second network model; wherein the second network model includes at least one second network substructure corresponding to each first network substructure, and each second network substructure includes one of the following: a stage structure, a block structure, and a processing layer.

[0177] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0178] It should be noted that, in the embodiments of this application, if the above-described model quantization method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0179] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in the above-described method.

[0180] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0181] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0182] It should be noted that the descriptions of the above-described storage media, computer program products, and device embodiments are similar to the descriptions of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the storage media, computer program products, and devices of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0183] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0184] Processor 701 typically controls the overall operation of computer device 700.

[0185] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0186] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0187] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0188] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0190] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0192] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0193] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0194] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A model quantization method, characterized in that, The method includes: Obtain first output data of at least one first network substructure in a first network model; wherein each first output data is obtained by processing a calibration dataset using the first network model, and the calibration dataset includes at least one of the following: image data, point cloud data, and speech data; Using a second network model, the calibration dataset is processed based on the activation quantization flag of at least one second network substructure in the second network model to obtain the second output data of each second network substructure; wherein, the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized. For each of the first network substructures, the parameters of the second network substructure are adjusted based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model. If the preset conditions are met, the adjusted second network model will be determined as the third network model; In the process of processing the calibration dataset using the second network model, the activation quantization identifier of the second network substructure is randomly assigned based on a set probability distribution parameter; the activation quantization identifier of the second network substructure includes a first identifier representing the quantization of the activation value of the second network substructure, or a second identifier representing the non-quantization of the activation value of the second network substructure.

2. The method according to claim 1, characterized in that, The step involves using a second network model and processing the calibration dataset based on the activation quantization flags of at least one second network substructure within the second network model to obtain second output data for each second network substructure, including: The calibration dataset is determined as the input data for the first second network substructure in the second network model; For each second network substructure in the second network model, the input data of the second network substructure is processed based on the weight value and activation quantization flag of the second network substructure to obtain the second output data of the second network substructure, and the second output data is used as the input data of the next second network substructure.

3. The method according to claim 2, characterized in that, The second output data of the second network substructure is obtained by processing the input data of the second network substructure based on the weight values ​​and activation quantization flags of the second network substructure, including: Based on the rounding method of the weight values ​​in the second network substructure and the first quantization parameter, the weight values ​​of the second network substructure are quantized to obtain the quantized weight values. Based on the quantized weight values ​​and the activation quantization flag of the second network substructure, the input data of the second network substructure is processed to obtain the second output data of the second network substructure.

4. The method according to claim 3, characterized in that, When the activation quantization flag of the second network substructure is the first flag, during the processing of the input data of the second network substructure, the activation value in the second network substructure is quantized based on the second quantization parameter. And / or, if the activation quantization flag of the second network substructure is the second flag, the activation value in the second network substructure is not quantized during the processing of the input data of the second network substructure.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: For each second network substructure in the second network model, the activation quantization flag of the second network substructure is randomly assigned based on the set probability distribution parameters.

6. The method according to claim 5, characterized in that, The random assignment of activation quantization flags to the second network substructure based on the set probability distribution parameters includes at least one of the following: When the probability distribution parameter includes the quantization probability, the activation quantization identifier of the second network substructure is randomly assigned a value based on the quantization probability; wherein, the quantization probability represents the probability that the activation quantization identifier of the second network substructure is assigned a first identifier, and the first identifier represents the quantization of the activation value of the corresponding second network substructure. When the probability distribution parameter includes the quantized inactivation probability, the activation quantization identifier of the second network substructure is randomly assigned a value based on the quantized inactivation probability; wherein, the quantized inactivation probability represents the probability that the activation quantization identifier of the second network substructure is assigned a second identifier, and the second identifier represents that the activation value of the corresponding second network substructure is not quantized.

7. The method according to any one of claims 1 to 4, characterized in that, The adjustment of the parameters of the second network substructure based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model includes: Based on the first output data and the second output data, determine the loss value of the second network substructure; Based on the loss value, the parameters of the second network substructure are adjusted.

8. The method according to claim 7, characterized in that, The parameters of the second network substructure include the rounding method used to quantize the weight values ​​in the second network substructure; The adjustment of the parameters of the second network substructure based on the loss value includes: Based on the loss value, the rounding method used to quantize the weight values ​​in the second network substructure is updated, so that during the processing of the calibration dataset, the weight values ​​in the second network substructure are quantized using a quantization function corresponding to the rounding method.

9. The method according to any one of claims 1 to 4, characterized in that, The determination that the preset conditions are met includes at least one of the following: If, based on each of the first output data and each of the second output data, the loss value of each of the second network substructures is determined to satisfy a preset loss constraint, then the preset condition is determined to be satisfied. If the number of adjustments to each of the second network substructures meets the preset number constraint, then the preset condition is determined to be satisfied.

10. The method according to any one of claims 1 to 4, characterized in that, The method further includes: For each first network substructure in the first network model, quantization is performed according to at least one bit width to obtain the second network model; wherein, the second network model includes at least one second network substructure corresponding to each first network substructure, and each second network substructure includes one of the following: stage structure, block structure, and processing layer.

11. A model quantization device, characterized in that, include: The first acquisition module is used to acquire first output data of at least one first network substructure in the first network model; wherein each first output data is obtained by processing a calibration dataset using the first network model, and the calibration dataset includes at least one of the following: image data, point cloud data, and speech data; The processing module is used to process the calibration dataset using a second network model and based on the activation quantization flag of at least one second network substructure in the second network model to obtain second output data for each second network substructure; wherein the second network model is obtained by quantizing the first network model, and the activation quantization flag of each second network substructure indicates whether the activation value of the second network substructure is quantized. The adjustment module is used to adjust the parameters of the second network substructure for each of the first network substructures, based on the first output data of the first network substructure and the second output data of the second network substructure corresponding to the first network substructure in the second network model. The first determining module is used to determine the adjusted second network model as the third network model when it is determined that the preset conditions are met. In the process of processing the calibration dataset using the second network model, the activation quantization identifier of the second network substructure is randomly assigned based on a set probability distribution parameter; the activation quantization identifier of the second network substructure includes a first identifier representing the quantization of the activation value of the second network substructure, or a second identifier representing the non-quantization of the activation value of the second network substructure.

12. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein when read and executed by a computer, the computer program implements the steps of the method of any one of claims 1 to 10.

Images