Method for improving similarity of post-quantization model
By stating and adjusting the quantization parameters of each layer of the deep learning model, and using an asymmetric quantization method, the problem of large quantization loss in the existing technology is solved, and the model similarity and application effect are improved.
Patent Information
- Application Number
- CN202311737812.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-18
- Publication Date
- 2025-06-20
AI Technical Summary
When existing post-training quantization methods deal with models with special weights or feature distributions, conventional quantization may bring great losses, and it is difficult to effectively reduce losses through methods such as deviation calibration.
By counting the maximum and minimum values of the quantized feature distribution range of each layer, and combining the adjustment of the quantization weight, asymmetric quantization methods and loop searches are used to optimize the quantization parameters to reduce the quantization loss.
It effectively reduces the loss of the quantitative model, improves the similarity between the quantitative model and the original model, and reduces the accuracy error and effect deviation when applying the model.
Smart Images
Figure CN120181145A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of deep learning model quantization, and particularly relates to a method for improving the similarity of post-quantization models. Background Art
[0002] With the development of technology, in the past decade, neural networks have demonstrated powerful capabilities and vitality in the field of artificial intelligence (AI). To deploy AI algorithms to terminal devices, full-precision (floating-point) models are often not very friendly to hardware. For example, some hardware accelerators such as DSP / NPU support int8, and the floating-point operation speed is slow during low-power operation, so 8-bit quantization is required. Therefore, it is necessary to quantize the model. Model quantization refers to converting the floating-point algorithm of the neural network into a fixed-point one. The main significance of model quantization is to accelerate the inference speed on the model side, reduce device power consumption, and reduce storage space. The benefits brought by quantization can reduce the model size and accelerate the inference speed, but the disadvantage is that quantization will introduce quantization errors and reduce the model effect. Therefore, minimizing errors as much as possible is the primary task of model quantization.
[0003] However, the defects in the prior art are as follows:
[0004] Although current post-training quantization methods have been adapted to many models, for some models with relatively special weight or feature distributions, a conventional quantization may cause relatively large losses. In existing methods, when the losses are relatively large and one does not want to increase the quantization bit width (generally increasing the quantization bit width can reduce model quantization losses), performing bias calibration on the quantized feature maps and adding the standard deviation of the feature differences to the model weights for bias correction still does not work well for some models. Summary of the Invention
[0005] To solve the above problems, the purpose of the present application is: to further compensate for losses, this method provides a method that can better quantize model weights to reduce model losses and improve model similarity (i.e., the contrast between the original unquantized model and the quantized model). This method is a tuning strategy for deep learning model quantization. Since directly quantizing the model usually causes a certain amount of loss, resulting in accuracy errors or effect deviations from the original unquantized model, this method can be used to reduce quantization losses when the quantization loss error is relatively obvious. The model processed by quantization can only be deployed and run on the chip terminal, and it is applicable to all fields using deep learning technology applications, including image recognition, video processing, video detection, etc.
[0006] Starting point: The quantization feature distribution range of each layer of the impression model is not only the statistically maximum and minimum values, but also the quantization weights of the corresponding layer. Adjusting the quantization weights can also correct the feature distribution to a certain extent, so that the feature distribution and the original model feature distribution further reduce losses.
[0007] Specifically, the present invention provides a method for improving the similarity of a post-quantization model, and the method includes the following steps:
[0008] S1. Save the features of the original model: Save the features of each layer of the original model for calculating the similarity with the quantization model.
[0009] S2. Statistically analyze the feature quantization parameters and quantize the features: Through step S1, the feature results of each layer of the original model are obtained; statistically analyze its distribution range to obtain the maximum and minimum values; and use the asymmetric quantization method to quantize the feature results.
[0010] S3. Save the input features of the current layer of the quantization model: The inference process of the model is executed sequentially from beginning to end. Similarly, the quantization process also performs quantization inference from beginning to end, and saves the results of the quantized layers for inferring the quantization results of the current quantization layer.
[0011] S4. Quantize the weights: Applicable to the convolution operator, the weight data dimension is (convolution kernel width, convolution kernel height, number of input channels, number of output channels); further including:
[0012] S4.1 Obtain the maximum and minimum values of the weights:
[0013] Directly statistically analyze to obtain the maximum and minimum values of the weights, that is, obtain the minimum value weight_min_value and the maximum value weight_max_value.
[0014] S4.2 Quantize the weights channel by channel:
[0015] Wherein, Weight represents the weight data;
[0016] Obtain the value with the largest absolute value, denoted as
[0017] Abs_max = max(abs(weight_min_value), abs(weight_max_value));
[0018] Traverse the weights channel by channel, denoted as: for i in the number of input channels (traverse the weights channel by channel);
[0019] Obtain the value of the current channel, denoted as: weight_temp = Weight[:, :, :, i];
[0020] Initialize the similarity score, denoted as: best_score = -1000000;
[0021] Initialize the maximum value, denoted as: Abs_max_result = 0;
[0022] Loop the maximum value 32 times, increasing it by a little each time. Let the increase be 0.001. This is equivalent to magnifying the maximum value of the quantization parameter for the weights, expressed as:
[0023] For j in range(32):
[0024] num = j * 0.001
[0025] Abs_max = abs_max + num;
[0026] Here, the similarity score between the calculation result of the quantized weight and the original feature result will be calculated. The higher the score, the higher the similarity, and the maximum value is 1. Expressed as:
[0027] score = compute_score(float_feature, quantize_feature, weight, Abs_max, i)
[0028] If score > best_score, expressed as If score > best_score, then:
[0029] Save the highest score, expressed as: Best_score = score;
[0030] Save the current abs_max value, expressed as: Abs_max_result = abs_max;
[0031] Loop the maximum value 32 times, decreasing it by a little each time. Let the decrease be 0.001. This is equivalent to reducing the maximum value of the quantization parameter for the weights, expressed as:
[0032] For j in range(32):
[0033] num = j * 0.001
[0034] Abs_max = abs_max - num;
[0035] Here, the similarity score between the calculation result of the quantized weight and the original feature result will be calculated. The higher the score, the higher the similarity, and the maximum value is 1. Expressed as:
[0036] score = compute_score(float_feature, quantize_feature, weight, Abs_max, i)
[0037] If score > best_score, expressed as: If score > best_score, then:
[0038] Save the highest score, denoted as: Best_score = score;
[0039] Save the current abs_max value, denoted as: Abs_max_result = abs_max;
[0040] After the above loop is completed, the highest score is obtained, corresponding to the abs_max value of the highest score;
[0041] Then complete the quantization of the channel weights here
[0042] Bit_s = 2 ** bitwidth
[0043] Q_weight = round(weight_temp * (bit_s / abs_max_result))
[0044] If Q_weight < -bit_s, then Q_weight = -bit_s
[0045] If Q_weight > bit_s - 1, then q_weight = bit_s - 1
[0046] Quantize the i-th channel, denoted as: Weight[:, :, :, i] = Q_weight;
[0047] Among them, compute_score is to calculate the similarity score;
[0048] S5. Save the weight quantization parameters
[0049] Finally, uniformly save the abs_max corresponding to the highest score of each score in step S3; that is the parameter of the final quantization weight.
[0050] In step S2, when statistically calculating its distribution range to obtain the maximum and minimum values, it is obtained by searching, and the judgment criterion for searching is lp_loss, which further includes the following:
[0051] 1). First, directly obtain the maximum and minimum values of the feature distribution;
[0052] 2). Quantize and dequantize the features through the current maximum and minimum values:
[0053] Quantization: round means rounding, pow(2, 8) means 2 to the 8th power. Let x be a floating-point value and x_int8 be the value quantized to 8 bits. Then x_int8 = round(x / (feature maximum value - feature minimum value) * (pow(2, 8) - 1));
[0054] Inverse quantization: That is, the reverse operation. Let x’ be the floating-point value after inverse quantization, and x’ = x_int8 / (pow(2, 8) - 1) * (feature maximum value - feature minimum value);
[0055] 3). Calculate the lp_loss error. The smaller the error, the smaller the quantization loss; abs represents taking the absolute value. Lp_loss = pow(abs(x - x’), 2.4);
[0056] 4). According to the comparison of the magnitudes of lp_loss, find the feature maximum and minimum values when the loss is small;
[0057] 5). The above steps 2) to 4) are looped 80 times, and the maximum and minimum values each time are decreased and increased by 1%; the distribution range of the maximum and minimum values is shortened; finally, the feature maximum and minimum values with the smallest loss are searched according to lp_loss.
[0058] In the step S2, the adoption of the asymmetric quantization method further includes the following:
[0059] Quantization parameter calculation:
[0060] Among them, Round represents rounding;
[0061] bitwidth represents the quantization bit width;
[0062] input represents the input;
[0063] output represents the output;
[0064]
[0065]
[0066] Perform quantization:
[0067] Quant_range = 2 ** bitwidth
[0068] Output = round(input / delta) + Zero_point
[0069] If output < 0, then output = 0;
[0070] If output > quant_range, then output = quant_range.
[0071] In the step S3, it is necessary to infer the quantization result of the current quantization layer: Assume that the current layer performs convolutional layer inference, and the process is as follows:
[0072] 1) x_input is the quantization input of the current layer, weight is the weight quantized for the current layer, and bias is the bias of the current layer.
[0073] 2) Let output be the output of the current layer. The inference calculation formula is: ((x_input * weight) + bias).
[0074] In the step S4, the calculation of the similarity score includes calculating the cosine similarity between the original feature and the quantized feature; further including:
[0075] Implementation of compute_score:
[0076] Get the data (float_feature, quantize_feature, weight, Abs_max, i);
[0077] Among them, Float_feature is the original feature data;
[0078] Quantize_feature is the feature data input for this layer;
[0079] Weight is the weight;
[0080] abs_max is the quantization parameter, the maximum absolute value of the weight;
[0081] i is the number of channels;
[0082] Conv2d represents the convolution operation. The operation result can be obtained by inputting the feature and the weight; the first step is to quantize the weights of the corresponding channels:
[0083] Bit_s = 2 ** bitwidth
[0084] Q_weight = round(weight_temp * (bit_s / abs_max_result))
[0085] If Q_weight < -bit_s, then Q_weight = -bit_s
[0086] If Q_weight > bit_s - 1, then q_weight = bit_s - 1
[0087] Q_weight = Q_weight / bit_s * abs_max_result
[0088] Quantization of the i-th channel is expressed as: Weight[:, :, :, i] = Q_weight;
[0089]
[0090] The quantized feature Res is obtained and expressed as:
[0091] Res = Conv2d(quantize_feature, weight);
[0092] Calculate the cosine similarity:
[0093] dot represents matrix multiplication;
[0094] Sqrt represents the square root operation;
[0095] Sum represents the summation operation;
[0096] Unfold the data and remove the dimension, expressed as:
[0097] Float_feature = float_feature.reshape(-1);
[0098] Res = res.reshape(-1);
[0099] Score = dot(float_feature, res) / sqrt(sum(float_feature * float_feature) + sqrt(sum(res * res)));
[0100] Score is the similarity score.
[0101] Therefore, the advantage of this application is that: by calculating the quantization parameter through the scaling of the group channels of the weight, this method can obtain a set of more suitable quantization weight methods, which can further improve the similarity of the quantization model. This method is a tuning strategy for deep learning model quantization. Because directly quantizing the model usually has a certain amount of loss, resulting in accuracy errors or performance deviations from the original unquantized model. This method can be used to reduce the quantization loss and improve the performance when the quantization loss error is obvious. Brief Description of the Drawings
[0102] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application, but do not limit the present invention.
[0103] Figure 1 It is a schematic flowchart of this method.
[0104] Figure 2 It is a schematic diagram of the code flow used in the embodiment of this method. Detailed Embodiment
[0105] To more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0106] This method proposes a method for improving the similarity of post-quantization models, as Figure 1 shown, including the following steps:
[0107] Step S1. Save the features of the original model:
[0108] Save the features of each layer of the original model for calculating the similarity with the quantization model. Here, the similarity is not calculated, but only the output features of each layer are saved.
[0109] Step S2. Statistically calculate the feature quantization parameters and quantize the features:
[0110] Through step S1, the feature results of each layer of the original model are obtained. Statistically calculate its distribution range to obtain the maximum and minimum values, which are used to quantize the feature results;
[0111] The distribution range of the said statistically calculated features is not directly obtaining its maximum and minimum values like step S4.1. Instead, it is obtained through search, and the judgment criterion for the search is lp_loss. The process is roughly as follows:
[0112] 1). First, directly obtain the maximum and minimum values of the feature distribution;
[0113] 2). Quantize and dequantize the features through the current maximum and minimum values.
[0114] Quantization: round means rounding, pow(2, 8) means 2 to the 8th power. Suppose x is a floating-point value and x_int8 is the value quantized into 8 bits. Then x_int8 = round(x / (feature maximum value - feature minimum value) * (pow(2, 8) - 1));
[0115] Dequantization: That is, the reverse operation. Suppose x' is the dequantized floating-point value. Then x' = x_int8 / (pow(2, 8) - 1) * (feature maximum value - feature minimum value);
[0116] 3). Calculate the lp_loss error. The smaller the error, the smaller the quantization loss. abs means taking the absolute value. Lp_loss = pow(abs(x - x'), 2.4);
[0117] 4). According to the comparison of the magnitudes of lp_loss, find the feature maximum and minimum values when the loss is small;
[0118] 5). The above steps 2) to 4) are looped through 80 times, and the maximum and minimum values each time are decreased and increased by 1%; the distribution range of the maximum and minimum values is shortened; finally, the maximum and minimum values of the features with the smallest loss are searched according to lp_loss;
[0119] The specific quantization method is as follows: The asymmetric quantization method is adopted:
[0120] Quantization parameter calculation
[0121] Round means rounding;
[0122] bitwidth represents the quantization bit width;
[0123] input represents the input; output represents the output;
[0124]
[0125]
[0126] Quantization:
[0127] Quant_range = 2 ** bitwidth
[0128] Output = round(input / delta) + Zero_point
[0129] If output < 0, then output = 0
[0130] If output > quant_range, then output = quant_range.
[0131] Step S3. Save the input features of the current layer of the quantization model:
[0132] The inference process of the model is executed sequentially, from beginning to end. Similarly, the quantization process also performs quantization inference from beginning to end, and the results of the quantized layers are saved for inferring the quantization results of the current quantization layer. Here is what needs to be inferred: Taking the inference process of the convolutional layer as an example for the current layer: 1). x_input is the quantized input of the current layer, weight is the quantized weight of the current layer, and bias is the bias of the current layer;
[0133] 2). Let output be the output of the current layer, and the inference calculation formula: ((x_input * weight) + bias).
[0134] Step S4. Quantize the weights: Applicable to convolutional operators, the weight data dimension is (convolution kernel width, convolution kernel height, number of input channels, number of output channels); further including:
[0135] S4.1 Obtain the maximum and minimum weights:
[0136] Directly calculate the maximum and minimum weights, and denote the minimum value as weight_min_value and the maximum value as weight_max_value;
[0137] S4.2 Quantize the weights channel by channel:
[0138] Weight represents the weight data;
[0139] Obtain the value with the largest absolute value, denoted as:
[0140] Abs_max = max(abs(weight_min_value), abs(weight_max_value))
[0141] Set the loop body as: for i in the number of input channels (traverse the weights channel by channel):
[0142] Obtain the current channel value, denoted as: weight_temp = Weight[:, :, :, i];
[0143] Initialize the similarity score, denoted as: best_score = -1000000;
[0144] Initialize the maximum value, denoted as: Abs_max_result = 0;
[0145] Loop the maximum value 32 times, and increase it by a little each time. Assume the increase is 0.001. This is equivalent to enlarging the maximum value of the quantization parameter of the weights, denoted as:
[0146] for j in range(32):
[0147] num = j * 0.001
[0148] Abs_max = abs_max + num;
[0149] Here, the similarity score between the calculation result of the quantized weights and the original feature result will be calculated. The higher the score, the higher the similarity, and the maximum value is 1. It is denoted as:
[0150] score = compute_score(float_feature, quantize_feature, weight, Abs_max, i). The higher the score, the better, and the maximum value is 1. Among them, the calculation process of compute_score will be further described later;
[0151] If score > best_score, then:
[0152] Save the highest score, which is expressed as: Best_score = score;
[0153] Save the current abs_max value Abs_max_result = abs_max;
[0154] Loop the maximum value 32 times and increase it by a little each time. This is equivalent to making the maximum value of the quantization parameter of the weight smaller, which is expressed as:
[0155] For j in range(32):
[0156] num = j * 0.001
[0157] Abs_max = abs_max - num;
[0158] Here, the similarity score between the calculation result of the quantized weight and the original feature result will be calculated. The higher the score, the higher the similarity, and the maximum value is 1. It is expressed as:
[0159] score = compute_score(float_feature, quantize_feature, weight, Abs_max, i)
[0160] The higher the score, the maximum value is 1. Among them, the calculation process of compute_score will be further described later;
[0161] If score > best_score, which is expressed as If score > best_score, then:
[0162] Save the highest score, which is expressed as: Best_score = score;
[0163] Save the current abs_max value, which is expressed as: Abs_max_result = abs_max;
[0164] After the above loop is completed, the highest score is obtained, corresponding to the abs_max value of the highest score;
[0165] Then complete the quantization of the weight of this channel here:
[0166] Bit_s = 2 ** bitwidth
[0167] Q_weight = round(weight_temp * (bit_s / abs_max_result))
[0168] If Q_weight < -bit_s, then Q_weight = -bit_s;
[0169] If Q_weight > bit_s - 1, then q_weight = bit_s - 1;
[0170] Quantize the i-th channel, expressed as: Weight[:, :, :, i] = Q_weight;
[0171] The above compute_score calculation process for calculating the similarity score includes: mainly completing the cosine similarity calculation of the original features and the quantized features:
[0172] Implementation of compute_score:
[0173] Obtain the data (float_feature, quantize_feature, weight, Abs_max, i)
[0174] Float_feature The original feature data;
[0175] Quantize_feature The feature data input to this layer;
[0176] Weight The weight;
[0177] abs_max The quantization parameter, the maximum absolute value of the weight;
[0178] i The number of channels;
[0179] Conv2d represents the convolution operation. The operation result can be obtained by inputting the feature and the weight. However, this calculation method is not the focus of this application;
[0180] The first step is to quantize the weights of the corresponding channels
[0181] Bit_s = 2 ** bitwidth;
[0182] Q_weight = round(weight_temp * (bit_s / abs_max_result));
[0183] If Q_weight < -bit_s, then Q_weight = -bit_s;
[0184] If Q_weight > bit_s - 1, then q_weight = bit_s - 1;
[0185] Q_weight = Q_weight / bit_s * abs_max_result;
[0186] Quantize the i channel, marked as: Weight[:,:,:,i] = Q_weight;
[0187] Obtain the quantized feature Res, expressed as: Res = Conv2d(quantize_feature, weight);
[0188] Calculate the cosine similarity:
[0189] dot represents matrix multiplication;
[0190] Sqrt represents the square root operation;
[0191] Sum represents the summation operation;
[0192] Unfold the data and remove the dimension, expressed as:
[0193] Float_feature = float_feature.reshape(-1);
[0194] Res = res.reshape(-1);
[0195] Score = dot(float_feature, res) / sqrt(sum(float_feature * float_feature) + sqrt(sum(res * res)));
[0196] Score is the similarity score.
[0197] Step S5. Save the weight quantization parameters
[0198] Finally, save the abs_max corresponding to the highest score of each score in step S3. That is the parameter of the final quantization weight.
[0199] Furthermore, according to the above steps, a specific example is as follows:
[0200] When encountering a deep learning model, after the post - quantization process, the obtained similarity scores are as follows
[0201] Operator name Similarity score (0 - 1.0)
[0202] The first layer conv2d 0.999
[0203] The second layer conv2d 0.958
[0204] The third layer conv2d 0.927
[0205] As can be seen from the above similarity scores, the similarity decreases significantly at the second layer, indicating that the quantization loss at this layer is relatively large. At this time, it is necessary to adjust its quantization parameters to see if the similarity can be effectively improved and the error can be reduced.
[0206] Step S1, save the output feature values of each layer of the model;
[0207] Step S2, count the maximum and minimum values of the output features of the second layer, and calculate the quantization parameters
[0208] The first layer: maximum value 0.484, minimum value -0.241
[0209] The second layer: maximum value 0.884, minimum value 0
[0210] The third layer: maximum value 0.468, minimum value -0.112
[0211] Bitwidth = 8
[0212]
[0213]
[0214] For the first layer, delta = 0.00284, zero_point = 84
[0215] For the second layer, delta = 0.00346, zero_point = 0
[0216] For the third layer, delta = 0.00227, zero_point = 49;
[0217] Step S3, quantize the features of all layers and save the quantized output
[0218] For the first layer, x1 represents the floating-point feature of the first layer, x1_q represents the quantized feature, clip means to clip the input to a specified range, clip(x, 0, 255) means the output range is 0 to 255
[0219] x1_q = clip(round(x1 / delta of the first layer) + zero_point of the first layer, 0, 255)
[0220] x1_q = (x1_q - zero_point of the first layer) * delta of the first layer
[0221] At this time, x1_q is the dequantized value, and the value range is still -0.241 to 0.484, but there will be a quantization error compared to x1
[0222] The quantized features of the second layer are expressed as x2_q, and the quantized features of the third layer are expressed as x3_q. The quantization process is the same as above.
[0223] Step S4, quantize weights
[0224] The first layer has good similarity, so we skip the first layer and directly quantize the weight of the second layer. The weight of the second layer is recorded as weight;
[0225] The dimension of the weight data is [3, 3, 16, 32], where 32 represents the number of output channels. The maximum and minimum values of each channel should be counted in turn to quantize the weight. The code flow is as follows Figure 2 shown.
[0226] After the final loop traversal, the weight quantization is completed, and the best_score obtained is 0.9989;
[0227] The maximum value of Weight is 3.4233;
[0228] At this point, the max of the quantized weight is saved, and the model is quantized and similarity calculated again. Let's take a look at the similarity of the quantized model:
[0229] First layer conv2d 0.999
[0230] Second layer conv2d 0.9989
[0231] The third layer conv2d 0.9992;
[0232] It can be seen that after the max adjustment of the second layer's quantization weight obtained by the above method, the similarity of model quantization is improved from 0.958 to 0.9989, which is a significant improvement in reducing the model quantization error.
[0233] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for improving the similarity of post - quantization models, characterized in that, The method includes the following steps: S1. Save the original model features: Save the features of each layer of the original model for calculating the similarity with the quantized model; S2. Statistically calculate the feature quantization parameters and quantize the features: Through step S1, the feature results of each layer of the original model are obtained; statistically calculate its distribution range to obtain the maximum and minimum values; use the asymmetric quantization method to quantize the feature results; S3. Save the input features of the current layer of the quantized model: The inference process of the model is executed sequentially from beginning to end. Similarly, the quantization process also performs quantization inference from beginning to end, and saves the results of the quantized layers for inferring the quantization results of the current quantized layer; S4. Quantize the weights: Applicable to the convolution operator, the weight data dimension is (convolution kernel width, convolution kernel height, number of input channels, number of output channels); further includes: S4.1 Obtain the maximum and minimum values of the weights: Directly statistically calculate the maximum and minimum values of the weights, that is, obtain the minimum value weight_min_value and the maximum value weight_max_value; S4.2 Quantize the weights channel by channel: Where, Weight represents the weight data; Obtain the value with the largest absolute value, denoted as Abs_max = max(abs(weight_min_value), abs(weight_max_value)); Traverse the weights channel by channel, denoted as: for i in the number of input channels (traverse the weights channel by channel); Obtain the value of the current channel, denoted as: weight_temp = Weight[:, :, :, i]; Initialize the similarity score, denoted as: best_score = -1000000; Initialize the maximum value, denoted as: Abs_max_result = 0; Loop the maximum value 32 times and increase it by a little each time, assuming an increase of 0.001; which is equivalent to magnifying the maximum value of the quantization parameter of the weights, denoted as: For j in range(32): num = j * 0.001 Abs_max = abs_max + num; Here, the similarity score between the calculation result of the quantized weights and the original feature result will be calculated. The higher the score, the higher the similarity, and the maximum value is 1; denoted as: score = compute_score(float_feature, quantize_feature, weight, Abs_max, i) If score > best_score, denoted as If score > best_score, then: Save the highest score, denoted as: Best_score = score; Save the current abs_max value, denoted as: Abs_max_result = abs_max; Loop the maximum value 32 times and decrease it by a little each time, assuming a decrease of 0.001, which is equivalent to shrinking the maximum value of the quantization parameter of the weights, denoted as: For j in range(32): num = j * 0.001 Abs_max = abs_max - num; The similarity score between the calculation result of the quantized weight and the original feature result will be calculated here. The higher the score, the higher the similarity, and the maximum value is 1. It is expressed as: score = compute_score(float_feature, quantize_feature, weight, Abs_max, i) If score > best_score, it is expressed as: If score > best_score, then: Save the highest score, which is expressed as: Best_score = score; Save the current abs_max value, which is expressed as: Abs_max_result = abs_max; After the above loop is completed, the highest score and the corresponding abs_max value of the highest score are obtained; Then complete the quantization of the weights of this channel here Bit_s = 2 ** bitwidth Q_weight = round(weight_temp * (bit_s / abs_max_result)) If Q_weight < -bit_s, then Q_weight = -bit_s If Q_weight > bit_s - 1, then q_weight = bit_s - 1 Quantize the i-th channel, which is expressed as: Weight[:, :, :, i] = Q_weight; Among them, compute_score is to calculate the similarity score; S5. Save the weight quantization parameters Finally, uniformly save the abs_max corresponding to the highest score of each score in step S3; That is the parameter of the final quantized weight.
2. A method for improving the similarity of a post - quantization model according to claim 1, characterized in that, The step of obtaining the maximum and minimum values by statistically its distribution range in step S2 is obtained by searching. The judgment criterion for searching is lp_loss, which further includes the following: 1). First directly obtain the maximum and minimum values of the feature distribution; 2). Quantize and dequantize the features through the current maximum and minimum values: Quantization: round means rounding, pow(2, 8) means 2 to the 8th power. Let x be the floating-point value and x_int8 be the value quantized to 8 bits. Then x_int8 = round(x / (feature maximum value - feature minimum value) * (pow(2, 8) - 1)); Dequantization: That is, the reverse operation. Let x' be the dequantized floating-point value, x' = x_int8 / (pow(2, 8) - 1) * (feature maximum value - feature minimum value); 3). Calculate the lp_loss error. The smaller the error, the smaller the quantization loss; abs means taking the absolute value Lp_loss = pow(abs(x - x'), 2.4); 4). According to the comparison of the magnitudes of lp_loss, find the feature maximum and minimum values when the loss is small; 5). The above steps 2) to 4) are looped 80 times, and the maximum and minimum values each time are decreased and increased by 1%; the distribution range of the maximum and minimum values is shortened; finally, the maximum and minimum values of the feature with the smallest loss are searched according to lp_loss.
3. A method for improving the similarity of a post - quantization model according to claim 1, characterized in that, In the step S2, the adoption of the asymmetric quantization method further includes the following: Quantization parameter calculation: Among them, Round means rounding; bitwidth represents the quantization bit width; input represents the input; output represents the output; Perform quantization: Quant_range = 2 ** bitwidth Output = round(input / delta) + Zero_point If output < 0, then output = 0; If output > quant_range, then output = quant_range.
4. A method for improving the similarity of a post - quantization model according to claim 1, characterized in that, In the step S3, it is necessary to infer the quantization result of the current quantization layer: Assume that the current layer performs convolution layer inference, and the process is as follows: 1) x_input is the quantization input of the current layer, weight is the quantization weight of the current layer, and bias is the bias of the current layer 2) Let output be the output of the current layer, and the inference calculation formula: ((x_input * weight) + bias).
5. A method for improving the similarity of a post - quantization model according to claim 1, characterized in that, In the step S4, the calculation of the similarity score includes completing the cosine similarity calculation of the original feature and the quantized feature; further including: Implementation of compute_score: Get the data (float_feature, quantize_feature, weight, Abs_max, i); Among them, Float_feature is the original feature data; Quantize_feature is the feature data input to this layer; Weight is the weight; abs_max is the quantization parameter, the maximum absolute value of the weight; i is the number of channels; Conv2d represents the convolution operation, and the operation result can be obtained by inputting the feature and the weight; The first step is to quantize the weights of the corresponding channels: Bit_s = 2 ** bitwidth Q_weight = round(weight_temp * (bit_s / abs_max_result)) If Q_weight < -bit_s, then Q_weight = -bit_s If Q_weight > bit_s - 1, then q_weight = bit_s - 1 Q_weight = Q_weight / bit_s * abs_max_result Quantization of the i channel is expressed as: Weight[:, :, :, i] = Q_weight; The quantized feature Res is obtained, expressed as: Res = Conv2d(quantize_feature, weight); Calculate the cosine similarity: dot represents matrix multiplication; Sqrt represents the square root operation; Sum represents the summation operation; Expand the data and remove the dimension, expressed as: Float_feature = float_feature.reshape(-1); Res = res.reshape(-1); Score = dot(float_feature, res) / sqrt(sum(float_feature * float_feature) + sqrt(sum(res * res))); Score is the similarity score.