Model quantification method and system based on stability scoring

Abstract

The invention discloses a model quantification method and system based on stability scoring, and the method comprises the steps: calculating the output features of each layer of an original model and the probability distribution of the output features of each layer of a quantification model, calculating the stability value of each layer of the quantification model according to the probability distribution, and evaluating the stability of each layer of the model through the stability value. Bit width is distributed according to characteristics of each layer, the method does not depend on labels, is only based on network forward activation information, and simultaneously measures output distribution difference and direction consistency before and after quantization, and a hierarchical stability measurement index can be constructed according to a stability value to accurately identify a steady-state layer and a high-sensitivity layer for guiding precision distribution. The bit width is adaptively reduced by adopting an iterative search strategy on the basis of ascending sorting of the stability value scores; on the premise that precision and resource constraints are met, the model compression ratio is increased to the maximum extent; retraining is not needed in the whole process, and large-scale network one-key quantitative deployment is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of model quantification, and relates to a model quantification method and system based on stability scoring. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), deep neural networks have been maturely applied to various computer vision tasks, achieving high prediction accuracy. However, because deep learning models typically contain a large number of parameters that consume significant storage space and require significant computing resources during inference, they are difficult to deploy on resource-constrained embedded or edge devices. Consequently, researchers have conducted extensive research on methods for compressing and accelerating deep convolutional neural network models. Existing model compression methods include model quantization, network pruning, knowledge distillation, and low-rank decomposition.

[0003] Model quantization methods are now widely used for fast inference and deployment of deep neural network models. Post-training quantization (PTQ) has attracted considerable attention from researchers due to its flexibility, minimal performance loss, and the elimination of model retraining. However, existing post-training quantization methods often rely on theoretical assumptions or fixed bit widths for network layers during the quantization process, resulting in significant performance losses in the quantized networks.

[0004] Model quantization technology typically involves converting full-precision model (FP32) parameters into low-order integers (INT8, INT4, INT2, etc.) using a quantization method. This quantized model can significantly reduce computational complexity and storage requirements, improving overall inference speed. Model quantization technology can meet edge computing needs and optimize energy efficiency, enabling the efficient deployment of complex deep learning models on mobile devices and IoT devices such as smartphones, watches, tablets, smart homes, and sensors. Research and application of model quantization technology has made significant progress in both academia and industry. For example, quantization white papers released by companies such as Google systematically explain the model quantization process and principles. Extensive experiments and research have demonstrated the significant success of model quantization in accelerating both model training and inference. Model quantization deployment frameworks such as Qualcomm's AIMET (AI Model Efficiency Toolkit) and NVIDIA's TensorRT enable faster and more stable deployment of quantized models in industries such as autonomous driving and intelligent transportation.

[0005] Most recent quantization methods are post-training quantization, which is close to industrial applications. In AdaRound, the Taylor expansion method is used to analyze the loss changes caused by quantization, and a learnable rounding method is proposed to reconstruct the features of the pre-trained model. Li Yuhang et al. broke the limit of post-training quantization through block reconstruction. They believed that the AdaRound method had large weight perturbations. Therefore, they used the Gauss-Newton matrix to analyze the second-order Hessian matrix error and proposed the BRECQ quantization method using block reconstruction to further improve the accuracy of the post-training quantization model. In PDQuant, they found that existing quantization methods only considered local information when determining the quantization parameters and could not adjust the optimal quantization parameters. Therefore, they proposed a method that uses the difference in network prediction information before and after quantization to determine the quantization parameters, adjusted the activation analysis in PTQ, and alleviated the overfitting problem. However, for post-training quantization, which is manually or simply uses some prior knowledge to allocate the quantization bit width, it may be suboptimal and biased, and will cause certain losses to the overall performance of the model.

[0006] Because each layer in a neural network model has different parameter counts and weight distributions, and therefore different quantization sensitivities, optimizing the bit distribution space within each layer to assign different quantization bit widths to each layer may be a more optimal approach. Therefore, introducing mixed-precision quantization methods during post-training quantization can further improve network model performance. Among existing post-training mixed-precision quantization methods, the ZeroQ algorithm uses knowledge distillation to measure the quantization sensitivity of each layer in the network model and then selects the mixed-precision configuration that minimizes the overall quantization sensitivity under specific model constraints. However, this method assumes that the bit width selection of other layers does not affect the quantization sensitivity of different layers. The AdaQuant algorithm, on the other hand, allocates the quantization bit width of each layer by analyzing integer programming formulas. However, this mixed-precision quantization bit width allocation process also fails to consider the potential impact between layers in the network model. Therefore, deeper layers in deep neural networks are inevitably affected by other layers. Therefore, it is necessary to consider the relationship between each layer and select appropriate metrics to determine the quantization bit width of different layers. Although there are various mixed-precision quantization (MPQ) methods, the following shortcomings still exist: current mixed-precision quantization methods mainly allocate bits based on indicators such as layer-level accuracy contribution or mean square error, and lack a systematic evaluation of "post-quantization output stability". At the same time, most accuracy evaluations rely on task labels and feature fitting capabilities, which makes it difficult to reflect the impact of inter-layer perturbation transmission on overall performance. Summary of the Invention

[0007] The purpose of the present invention is to solve the problem that most post-training quantization methods in the prior art are based on theoretical assumptions or fixed allocation of the bit width of the network layer. The current mixed precision quantization method mainly allocates bits based on indicators such as layer accuracy contribution or mean square error, and lacks a systematic evaluation of the "post-quantization output stability". At the same time, most accuracy evaluations rely on task labels and feature fitting capabilities, which makes it difficult to reflect the impact of inter-layer disturbance transmission on overall performance. A model quantization method and system based on stability scoring is provided.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A model quantification method based on stability scoring includes the following steps:

[0010] Obtain the original model and the quantized model after quantizing the original model;

[0011] Input the image samples into the original model and the quantized model respectively, obtain the output features of each layer of the original model and the output features of each layer of the quantized model respectively, calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model, calculate the stability value of each layer of the quantized model based on the probability distribution, and reorder each layer of the quantized model based on the stability value;

[0012] Calculate the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model, set a reference threshold, compare the error with the reference threshold, and redetermine the bit width allocation of each layer of the quantization model based on the comparison result to obtain the model after quantization.

[0013] A further improvement of the present invention is:

[0014] The calculation of the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model includes:

[0015] Initialize the bits of all layers of the quantization model to high bit width INT8;

[0016] Traverse all convolutional layers and calculate the error ε between the output of each layer of the quantized model and the output of each layer of the original model at the current bit width.

[0017] Comparing the error with a reference threshold and re-determining the bit allocation of each layer of the quantization model according to the comparison result includes:

[0018] If the error ε is less than the reference threshold α, the bit width of the current layer is retained;

[0019] If the error ε is greater than or equal to the reference threshold α, the bit width is reduced layer by layer, and the error ε of each layer is recalculated until the errors of all layers meet the error ε less than the reference threshold α, and the bit allocation of the model is completed.

[0020] Calculating the stability value of each layer of the quantization model according to the probability distribution includes:

[0021] Calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model:

[0022]

[0023] in, Represents the output of each layer of the original model; Represents the output of each layer of the quantized model; Represents the activation distribution of the i-th layer of the original model; Represents the activation distribution of the i-th layer of the quantized model;

[0024] According to the probability distribution, the stability score of each layer is calculated:

[0025]

[0026] in, Represents the relative entropy between two activation distributions, characterizing the distribution shift caused by quantization; Cosine similarity calculates the directional consistency of the activation tensor and reflects the overall structural perturbation of the activation; λ1 and λ2 both represent weight factors.

[0027] The original model is subjected to PTQ quantization to obtain a quantized model.

[0028] The original model is a float32 model.

[0029] It also includes setting judgment conditions for the quantization model after the bit width is reallocated. The judgment conditions include model accuracy and model size. If the model accuracy is greater than or equal to 95% and the model size is less than or equal to 10MB, the quantization model meets the requirements. Otherwise, re-quantization is performed.

[0030] A model quantification system based on stability scoring, comprising:

[0031] A model acquisition module is used to obtain the original model and the quantized model after quantizing the original model;

[0032] A stability value evaluation module is used to input image samples into the original model and the quantized model respectively, obtain the output features of each layer of the original model and the output features of each layer of the quantized model respectively, calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model, calculate the stability value of each layer of the quantized model based on the probability distribution, and reorder each layer of the quantized model based on the stability value;

[0033] The model quantization module is used to calculate the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model, set a reference threshold, compare the error with the reference threshold, and redetermine the bit width allocation of each layer of the quantization model based on the comparison result to obtain the model after quantization.

[0034] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0035] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any method described in the present invention when executed by a processor.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This invention discloses a model quantization method based on stability scoring. The method calculates the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model. Based on the probability distribution, the stability value of each layer of the quantized model is calculated. The stability value is used to evaluate the stability of each layer of the model. Bit widths are allocated based on the characteristics of each layer, independent of labels and based solely on network forward activation information. The method also measures the difference in output distribution before and after quantization and directional consistency. A hierarchical stability metric is constructed based on the stability value to accurately identify "steady-state layers" and "highly sensitive layers" to guide precision allocation. The stability score is sorted in ascending order, and an iterative search strategy is used to adaptively reduce bit widths. This method maximizes the model compression ratio while meeting precision and resource constraints. The entire process does not require retraining, supporting one-click quantization deployment of large-scale networks. The proposed QSS stability score, as an independent unit, can be inserted into any PTQ framework. The original model structure does not need to be modified, resulting in strong system adaptability. This method, which does not rely on training or label information, accurately captures the perturbation contribution of each layer to precision, fully leveraging the model's performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a schematic diagram of an improved system structure module according to an embodiment of the present invention;

[0040] Figure 2 This is a flow chart of stability scoring according to an embodiment of the present invention;

[0041] Figure 3 This is a comparison chart of the accuracy and compression ratio of MobileNetV2 under different quantization strategies in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0045] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0047] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0048] The present invention is described in further detail below with reference to the accompanying drawings:

[0049] See also Figures 1 to 3 The present invention discloses a model quantization method based on stability scoring. Forward inference is performed on both the original full-precision model (float32) and a uniform bit-width quantization model (e.g., INT8), collecting the output activation tensors of each layer. Subsequently, based on the distribution differences (e.g., KL divergence) and directional consistency (e.g., cosine similarity) between these tensors, a stability score (QSS) is calculated for each network layer to measure its sensitivity to quantization perturbations. The bit width of each layer is gradually reduced in descending order of stability. After each adjustment, the error value ε between the quantized output and the original output of the layer under the current configuration is calculated in real time and compared with a preset tolerance threshold α to determine whether the current bit width is acceptable. If ε exceeds the threshold, the layer falls back to the previous bit width to ensure accuracy safety. On this basis, a mixed-precision bit configuration scheme is ultimately generated that meets the accuracy target (e.g., Top-1 ≥ 95%) and resource constraints (e.g., model size ≤ 10MB). This method does not rely on the training process and does not require label information. It only constructs a scoring system based on the activation change pattern to accurately capture the perturbation contribution of each layer to the accuracy. This method proposes:

[0050] Stability Scoring Mechanism (QSS): This mechanism does not rely on labels and is based solely on network forward activation information. It simultaneously measures the difference in output distribution and directional consistency before and after quantization, constructs a hierarchical stability metric to accurately identify "steady-state layers" and "highly sensitive layers" to guide precision allocation.

[0051] Mixed-precision bit width allocation strategy: Based on ascending sorting of QSS scores, an iterative search strategy is used to adaptively reduce the bit width; while meeting accuracy and resource constraints, the model compression ratio is maximized; the entire process does not require retraining, supporting one-click quantization deployment of large-scale networks.

[0052] Modular system architecture: QSS modules are independent units that can be plugged into any PTQ framework. The original model structure does not need to be modified, and the system is highly adaptable.

[0053] The acquisition of the QSS score in the embodiment of the present invention includes:

[0054] Get the output feature (activation) distribution of each layer before and after quantization;

[0055] Use KL divergence and cosine similarity to construct stability score;

[0056] All layers are ranked based on the scores and the bit widths are allocated iteratively.

[0057] The input module of this method receives the inference sample input (usually unlabeled image data) for model inference, and feeds the input data to the full-precision model and the quantized model for forward propagation; the full-precision model serves as a reference model and outputs the unquantized activation data of each layer, which can be any deep neural network (such as ResNet, MobileNet, ViT), and outputs the layer activation tensor The activation data is then entered into the QSS model for scoring; after training, the quantization model uses traditional PTQ tools such as (OMSE, AdaRound, BRECQ) to quantize the full-precision model layer by layer, and the accuracy level remains consistent throughout the text (NT8 / INT6 / INT4 / INT2), and outputs the quantized activation tensor The activation data then enters the QSS module and is compared with the full-precision activation; the input of the stability scoring module (QSS) is the activation pair By constructing the distribution, calculating the KL divergence + the direction difference of the active vector to comprehensively calculate the stability score, and sending the score result to the bit width configuration module; the bit width configuration module prioritizes reducing the bit width of the stability layer according to the score, sorting the levels in ascending order, iteratively searching for the optimal compression ratio configuration, and obtaining the mixed precision configuration solution.

[0058] The specific steps include:

[0059] Step 1: Layer output sampling: Obtain the response output of each layer in the model in full precision and quantized states for subsequent quantitative perturbation evaluation.

[0060] Step 1.1: Input the original test image sample (no label required);

[0061] Step 1.2: Perform forward inference on each network layer i and collect the output tensor:

[0062] Full-precision model (FP32) output activations In this step, the full-precision model is the original model, and the original model is a float32 model, that is, a 32-bit floating-point model.

[0063] Quantized model (INTx) output activation: In this step, the quantized model is the model after PTQ quantization of the original model. In this step, PTQ refers to Post-Training Quantization.

[0064] Step 1.3: Output each layer activation pair (floating point / quantized)

[0065] At the same time, let the original float32 model (denoted as FP) and the quantized int model (denoted as INT) run a batch of images; then save the output results of a certain layer, which are: Represents the output of the i-th layer of the full-precision model; Represents the output of the i-th layer of the quantized model.

[0066] Furthermore, these output tensors are normalized and histograms are calculated to obtain two distributions: The specific acquisition process is in step 2.

[0067] Step 2: Activation distribution fitting and stability score calculation (QSS)

[0068] During the quantization process, although the output accuracy may vary within an acceptable range, if the output distribution shifts too much after quantization, or the activation direction is unstable, it is easy to cause the cumulative error of subsequent layers to be amplified. This paper introduces the QSS indicator to compare and analyze the quantization perturbation before and after quantization, and comprehensively evaluate the "stability" of each layer. Specifically, it includes:

[0069] Step 2.1: Distribution Build

[0070] Activation tensor Perform distribution fitting. Use the histogram to estimate the probability distribution:

[0071]

[0072] in, Activation distribution of layer i of the floating-point model; Quantization model layer i activation distribution

[0073] Step 2.2: Stability score calculation formula:

[0074] Calculate the stability score (Quantization Stability Score, QSS) of each layer:

[0075]

[0076] in, Represents the relative entropy between two activation distributions, characterizing the distribution shift caused by quantization; Represents cosine similarity, calculates the directional consistency of the activation tensor, and reflects the overall structural perturbation of the activation; λ1 and λ2 represent weight factors, and the default empirical setting is 0.6 / 0.4.

[0077] Step 2.3: Output the stability score QSS of each layer i i .

[0078] Step 3: Stability sorting and bit width allocation strategy.

[0079] Step 3.1: Sorting and prioritizing

[0080] QSS for all layers i i Scores are sorted in ascending order (the lower the score, the more stable it is after quantification);

[0081] Then sort the QSS values ​​of all layers: those with high qualitative properties (small QSS) can use low bit width, and those with low stability (large QSS) can use high bit width.

[0082] Furthermore, the ones with the highest mark ranking are “low-sensitivity layers”, and a lower bit width (such as INT2, INT4) may be considered.

[0083] Step 3.2: Adjust the bit width in batches according to the sorting results 4

[0084] The process includes:

[0085] Step 3.2.1: Initialize the quantization model to have a high bit width of INT8 for all layers.

[0086] Initialize all layers to high bit width INT8: Initially, setting the bit width of all convolutional layers to 8-bit integers (INT8) is equivalent to the most "conservative" initial state;

[0087] Step 3.2.2: Traverse all convolutional layers and calculate the error ε at the current bit;

[0088] For each layer, a forward pass is run using the currently set bit width. The difference between the quantized output of this layer and the original model output is then compared. This difference is the error ε, which measures how different the layer is from the current compression. The error ε refers to the similarity between the two output tensors, Afp and Aint, and is measured using KL divergence (comparing the difference in the "probability distribution" of the two activations) and cosine distance (comparing the "directions" of the two activation tensors). This measures the change in the statistical structure of the activations and the overall change in the features (blurring, rotation). This difference determines whether the bit width of this layer can be safely reduced.

[0089] Step 3.3: Set the threshold α and determine whether the error meets the conditions (for example, less than the threshold α). If not, fine-tune the bit width (INT8 → INT6 → INT4 → INT2).

[0090] If not, the bit width is fine-tuned (INT8 → INT6 → INT4 → INT2): the error ε is evaluated layer by layer, and the bits of the current layer are adjusted based on ε: INT8 → INT6 → INT4... If ε is less than a certain threshold α, the bit width is adjusted repeatedly (INT8 → INT6 → INT4 → INT2) until a compromise is found.

[0091] In practice, the threshold α is an empirical threshold, for example, KL divergence α = 0.02-0.05; in this embodiment, ε is the combined QSS value, and α is the maximum tolerance value of QSS estimated based on actual conditions. Some papers make α a "dynamic adjustment parameter" or use cross-validation to determine the optimal value): this indicates that the layer is compressed well, so the current bits are retained. If ε is too large: this indicates that the layer is sensitive to quantization and cannot be compressed too hard, it can be "backed off" a bit and given more bits (for example, from INT4 to INT6);

[0092] Iterate until all layers have found a bit width configuration that is both space-efficient and error-free. (The configuration is complete if the error ε is less than α, or if the overall Top-1 accuracy of the model does not drop significantly.)

[0093] Step 3.4: Iterate until all layers meet the accuracy-stability target (Top-1Acc), complete the hybrid configuration, and record the current configuration.

[0094] Step 4: Output the final configuration

[0095] Set the judgment conditions and output the final bit configuration to meet the following requirements:

[0096] 1) Meet the accuracy target (e.g., Top-1 accuracy of the model ≥ 95%). In this embodiment, Top-1 accuracy refers to the proportion of the most confident category predictions in the model output that are consistent with the true label. It is an important indicator for measuring the performance of the classification model and requires that the model's recognition ability cannot be poor and the task accuracy must be achieved, such as a classification accuracy ≥ 95%.

[0097] 2) Satisfy resource constraints (e.g., model size ≤ 10MB).

[0098] If all conditions are met, the current mixed precision configuration is output; otherwise, it is reverted. This configuration can be directly deployed to the quantization compiler to generate a deployable model.

[0099] If both conditions are met, the quantization is successful and the configuration can be output for deployment; otherwise, the configuration must be rolled back and the bit width readjusted.

[0100] Compared with the traditional post-training quantization method that uses a uniform bit width for compression, the hybrid precision quantization method based on the stability scoring mechanism proposed in this invention can accurately evaluate the sensitivity of each layer of the network to quantization perturbations without relying on training samples and label information, and on this basis allocate the optimal bit width to different layers, thereby effectively improving the synergistic performance of the overall model accuracy and compression efficiency.

[0101] In order to further verify the performance improvement of the network model after quantization of the proposed method, Figure 3 This paper compares the performance of fixed 8-bit quantization and the proposed QSS mixed-precision quantization strategy on the MobileNetV2 model. The experiments were conducted on a subset of the ImageNet validation set, using a uniform PyTorch post-training quantization pipeline and a uniform sample generation layer activation distribution.

[0102] from Figure 3 As can be seen in the figure, under the same test conditions, the QSS mixed-precision quantization model improved the Top-1 classification accuracy by 2.1 percentage points compared to the fixed 8-bit model. At the same time, the model compression ratio increased from 8.0 to 9.1, demonstrating improved storage efficiency and control over inference resource usage. This on-demand bit width adjustment mechanism not only improves accuracy retention but also enhances the model's adaptability to edge deployments.

[0103] In addition, the bit width configuration scheme generated by this method has good portability and can be directly applied to the inference deployment process of various lightweight neural networks (such as MobileNet, ShuffleNet, EfficientNet, etc.). Its model size, energy consumption control and real-time performance are further verified on the actual deployment platform, which reflects the significant advantages of the mixed precision quantization method in the field of model compression.

[0104] This embodiment also discloses a model quantification system based on stability scoring, including:

[0105] A model acquisition module is used to obtain the original model and the quantized model after quantizing the original model;

[0106] A stability value evaluation module is used to input image samples into the original model and the quantized model respectively, obtain the output features of each layer of the original model and the output features of each layer of the quantized model respectively, calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model, calculate the stability value of each layer of the quantized model based on the probability distribution, and reorder each layer of the quantized model based on the stability value;

[0107] The model quantization module is used to calculate the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model, set a reference threshold, compare the error with the reference threshold, and redetermine the bit width allocation of each layer of the quantization model based on the comparison result to obtain the model after quantization.

[0108] A schematic diagram of a terminal device provided in one embodiment of the present invention. The terminal device in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of each of the aforementioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the aforementioned device embodiments are implemented.

[0109] The computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to accomplish the present invention.

[0110] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0111] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0112] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0113] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0114] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A model quantification method based on stability scoring, characterized in that: The following steps are involved: Obtain the original model and the quantized model after quantizing the original model; Input the image samples into the original model and the quantized model respectively, obtain the output features of each layer of the original model and the output features of each layer of the quantized model respectively, calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model, calculate the stability value of each layer of the quantized model based on the probability distribution, and reorder each layer of the quantized model based on the stability value; Calculate the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model, set a reference threshold, compare the error with the reference threshold, and redetermine the bit width allocation of each layer of the quantization model based on the comparison result to obtain the model after quantization.

2. A model quantification method based on stability scoring according to claim 1, characterized in that: The calculation of the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model includes: Initialize the bits of all layers of the quantization model to high bit width INT8; Traverse all convolutional layers and calculate the error ε between the output of each layer of the quantized model and the output of each layer of the original model at the current bit width.

3. A model quantification method based on stability scoring according to claim 2, characterized in that: Comparing the error with a reference threshold and re-determining the bit allocation of each layer of the quantization model according to the comparison result includes: If the error ε is less than the reference threshold α, the bit width of the current layer is retained; If the error ε is greater than or equal to the reference threshold α, the bit width is reduced layer by layer, and the error ε of each layer is recalculated until the errors of all layers meet the error ε less than the reference threshold α, and the bit allocation of the model is completed.

4. The model quantification method based on stability scoring according to claim 1, characterized in that: Calculating the stability value of each layer of the quantization model according to the probability distribution includes: Calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model: in, Represents the output of each layer of the original model; Represents the output of each layer of the quantized model; Represents the activation distribution of the i-th layer of the original model; Represents the activation distribution of the i-th layer of the quantized model; According to the probability distribution, the stability score of each layer is calculated: in, Represents the relative entropy between two activation distributions, characterizing the distribution shift caused by quantization; Cosine similarity calculates the directional consistency of the activation tensor and reflects the overall structural perturbation of the activation; λ1 and λ2 both represent weight factors.

5. The model quantification method based on stability scoring according to claim 1, characterized in that: The original model is subjected to PTQ quantization to obtain a quantized model.

6. A model quantification method based on stability scoring according to claim 1, characterized in that: The original model is a float32 model.

7. The model quantification method based on stability scoring according to claim 1, characterized in that: It also includes setting judgment conditions for the quantization model after the bit width is reallocated. The judgment conditions include model accuracy and model size. If the model accuracy is greater than or equal to 95% and the model size is less than or equal to 10MB, the quantization model meets the requirements. Otherwise, re-quantization is performed.

8. A model quantification system based on stability scoring, characterized in that: include: A model acquisition module is used to obtain the original model and the quantized model after quantizing the original model; A stability value evaluation module is used to input image samples into the original model and the quantized model respectively, obtain the output features of each layer of the original model and the output features of each layer of the quantized model respectively, calculate the probability distribution of the output features of each layer of the original model and the output features of each layer of the quantized model, calculate the stability value of each layer of the quantized model based on the probability distribution, and reorder each layer of the quantized model based on the stability value; The model quantization module is used to calculate the error between the output features of each layer of the reordered quantization model and the output features of each layer of the original model, set a reference threshold, compare the error with the reference threshold, and redetermine the bit width allocation of each layer of the quantization model based on the comparison result to obtain the model after quantization.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images