A model quantization method for an edge computing platform

Abstract

The application provides a model quantization method for an edge computing platform, and relates to the technical field of machine learning, which comprises the following steps: iteratively pruning a model of the edge computing platform to obtain a candidate deployment model; processing an image data set by using the candidate deployment model to obtain activation values corresponding to each layer; calculating the activation values by using an activation function to obtain a quantization lower limit and a quantization upper limit; quantizing the candidate deployment model by using a low-bit quantization method to obtain a target deployment model, thereby completing the model quantization of the edge computing platform; and processing input task data of the edge computing platform by using the target deployment model to obtain a task processing result. The application solves the problems of complex model structure and large resource consumption of the edge computing platform.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a model quantization method for edge computing platforms. Background Technology

[0002] Edge computing platforms support a full range of visual AI tasks, including detection, segmentation, pose estimation, tracking, and classification. These platforms are high-performance, low-power embedded development boards designed for edge computing, artificial intelligence (AI), and deep learning inference applications. However, deploying models from edge computing platforms still presents significant challenges. The main issues are limited computing resources, poor real-time performance, and high energy consumption: edge computing devices lack high-end GPU computing platforms, and their number of computing cores is far fewer than that of desktop computers. Deep neural network structures are highly complex, and edge computing platforms employ complex models with numerous layers. The inference process requires a large number of convolutional operations, which are extremely resource-intensive. Furthermore, deep neural networks have a large number of parameters; each data point is in FP32 format, meaning each value occupies 4 bytes, resulting in a large amount of GPU memory required for deep neural network inference. Summary of the Invention

[0003] In view of the above-mentioned shortcomings in the prior art, the present invention provides a model quantization method for edge computing platforms, which solves the problems of complex model structure and high resource consumption of edge computing platforms.

[0004] To achieve the aforementioned objectives, the present invention employs the following technical solution: a model quantization method for edge computing platforms, comprising:

[0005] S1: The model of the edge computing platform is iteratively pruned step by step to obtain candidate deployment models; wherein, the candidate deployment models are used to process the image dataset to obtain the activation values ​​corresponding to each layer;

[0006] S2: Calculate the activation value using the activation function to obtain the minimum and maximum quantization cutoff values;

[0007] S3: Based on the minimum and maximum values ​​of the quantization cutoff, perform low-bit quantization on the candidate deployment model to obtain the target deployment model, thus completing the model quantization of the edge computing platform; wherein, the target deployment model is used to process the input task data of the edge computing platform to obtain the task processing result.

[0008] The beneficial effects of this invention are as follows: It provides a model quantization method for edge computing platforms. (1) It adopts a structured pruning method, which gradually eliminates redundant parameters and significantly reduces the model size while maintaining model accuracy as much as possible, thereby reducing the number of calculations in the model inference process. Then, it performs low-bit quantization to convert floating-point operations into fixed-point operations, reducing computational complexity. It performs lossless compression of the model from both structure and data format, and deploys it to the edge computing platform, thereby reducing memory usage and increasing inference speed. (2) It has a significant improvement in efficiency, adaptability, and real-time performance, achieving a good balance between model depth and performance, and improving the convenience of model deployment. The pruning-quantization joint optimization forms a complete process, reducing the need for manual parameter tuning. It supports one-click conversion of mainstream edge frameworks (TensorRT).

[0009] Further, S1 includes:

[0010] Based on the target pruning rate, the model of the edge computing platform is iteratively pruned step by step, resulting in the corresponding pruning accuracy loss:

[0011]

[0012] Where c represents the pruning accuracy loss, p represents the target pruning rate, and n represents the number of iterations;

[0013] When the pruning accuracy loss reaches a threshold, the iteration stops, and a candidate deployment model is obtained.

[0014] Furthermore, the expressions for the minimum and maximum quantization cutoff values ​​are as follows:

[0015] clip min =min(x);

[0016] clip max =max(x);

[0017] SiLU(x') = x'·σ(x');

[0018]

[0019] Among them, clip min This represents the minimum value of the quantization cutoff, min represents the minimum value function, x represents the activation value after activation, and clip represents the minimum value. max σ represents the maximum value of the quantization cutoff, max represents the maximum value function, SiLU represents the SiLU function, x' represents the activation value of the input SiLU, and σ represents the Sigmoid function.

[0020] Further, S2 includes:

[0021] By analyzing the activation values ​​corresponding to each layer, the range of activation values ​​for each layer is obtained;

[0022] Based on the range of activation values ​​for each layer, the minimum value of the activation value is fixed as the minimum value of the quantization cutoff. The maximum value of the activation value is traversed, and the distribution of the determination coefficients of each maximum cutoff value is obtained through calculation.

[0023] Based on the distribution of the determination coefficients, the maximum value among them is selected as the optimal determination coefficient;

[0024] Based on the optimal determination coefficient, the corresponding maximum truncation value is taken as the maximum quantization truncation value, thus obtaining the minimum quantization truncation value and the maximum quantization truncation value.

[0025] Furthermore, the expression for the determination coefficient is:

[0026]

[0027] Among them, R 2 The coefficient of determination is represented by x, which represents the activation value after activation. The quantization cutoff interval is represented by SSE, which represents the sum of squared residuals between the reference distribution and the quantization distribution. SST represents the total sum of squares between the reference distribution and the quantization distribution. quant_center represents the activation value, center represents the median of the interval, i represents the activation value index, count represents the number of activation values, and mean_center represents the mean of the original data.

[0028] Furthermore, the expressions for the minimum and maximum quantization cutoff values ​​are as follows:

[0029] clip min =min(x);

[0030] clip max =max(x);

[0031] SiLU(x') = x'·σ(x');

[0032]

[0033] Among them, clip min This represents the minimum value of the quantization cutoff, min represents the minimum value function, x represents the activation value after activation, and clip represents the minimum value. max σ represents the maximum value of the quantization cutoff, max represents the maximum value function, SiLU represents the SiLU function, x' represents the activation value of the input SiLU, and σ represents the Sigmoid function.

[0034] Furthermore, the minimum and maximum quantization cutoff values ​​include: further uniformly truncating activation values ​​outside the quantization cutoff range to obtain the corresponding minimum and maximum quantization cutoff values.

[0035]

[0036] Dynamic truncation overcomes the limitations of traditional fixed truncation ranges. Simultaneously, R is introduced. 2 As a benchmark, it offers a 30% improvement in accuracy compared to TensorRT's automatic 8-bit quantization. Attached Figure Description

[0037] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0038] Figure 1 This is an exemplary flowchart illustrating a model quantization method for an edge computing platform according to some embodiments of this specification. Detailed Implementation

[0039] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0040] Example

[0041] Figure 1 This is an exemplary flowchart illustrating a model quantization method for an edge computing platform according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process may be executed by a processor.

[0042] S1: The model of the edge computing platform is iteratively pruned step by step to obtain candidate deployment models; wherein, the candidate deployment models are used to process the image dataset to obtain the activation values ​​corresponding to each layer.

[0043] The model of an edge computing platform is a neural network model mounted on the edge computing platform. For example, the model of an edge computing platform can include deep neural network models, convolutional neural network models, and YOLOv8 models.

[0044] An edge computing platform is an embedded development board used for edge computing and deep learning, located close to the end user.

[0045] The candidate deployment model is a lightweighted model of the edge computing platform.

[0046] In some embodiments, the processor can construct dependencies for each layer of the edge computing platform's model. During pruning, it combines dependency group relationships to calculate the importance of weight metrics for each layer in each dependency group, including the sum of squares of the weights or the magnitude of their influence by the loss function. The weights within each group are ranked according to their influence, and the lowest-weighted subset is removed to obtain candidate deployment models. For example, the processor can construct dependency groups around a convolutional layer as the main body, removing the main body's output channels while keeping the convolutional kernel size unchanged, to obtain candidate deployment models.

[0047] In this way, the number of channels in the upstream and downstream layers of the dependency group remains consistent, avoiding the disruption of inter-layer dependencies. The generated model can be deployed directly without additional processing.

[0048] Image datasets include image data for detection, segmentation, pose estimation, tracking, and classification.

[0049] In some embodiments, the processor can input the image dataset to be detected into the candidate deployment model for processing, record the activation value of each layer in the inference of the candidate deployment model, and obtain the corresponding activation value.

[0050] In some embodiments, the processor may implement S1 based on the following steps: performing iterative pruning on the model of the edge computing platform based on the target pruning rate to obtain the corresponding pruning accuracy loss; when the pruning accuracy loss reaches a threshold, stopping the iteration to obtain a candidate deployment model.

[0051] The target pruning rate is the preset efficiency for each pruning.

[0052] In some embodiments, the processor can adjust the pruning efficiency for each pruning based on the pruning test results to obtain a target pruning rate. For example, the target pruning rate can be set to 0.5.

[0053] Pruning accuracy loss reflects the decrease in model accuracy of the edge computing platform after each pruning step. For example, a higher pruning accuracy loss indicates a lower decrease in model accuracy of the edge computing platform.

[0054] In some embodiments, the expression for the pruning accuracy loss can be:

[0055]

[0056] Where c represents the pruning accuracy loss, p represents the target pruning rate, and n represents the number of iterations.

[0057] In some embodiments, the processor can train and fine-tune the model after each pruning to restore performance. Pruning and fine-tuning are performed alternately to gradually reduce the model size while ensuring maximum accuracy recovery, resulting in a candidate deployment model. For example, a candidate deployment model with minimal accuracy loss can be obtained when the target pruning rate is 0.5 and the number of iterations is 20.

[0058] S2: Calculate the candidate deployment model using the activation function to obtain the minimum and maximum quantization cutoff values.

[0059] The minimum quantization cutoff value is the minimum value of the quantization range.

[0060] The maximum quantization cutoff value is the maximum value of the quantization range.

[0061] In some embodiments, the processor may implement S2 based on the following steps: by analyzing the activation values ​​corresponding to each layer, the range of activation values ​​for each layer is obtained; based on the range of activation values ​​for each layer, the minimum value of the activation value is fixed as the minimum quantization cutoff value, the maximum value of the activation value is traversed, and the distribution of determination coefficients for each maximum cutoff value is obtained by calculation; based on the distribution of determination coefficients, the maximum value among them is selected as the optimal determination coefficient; based on the optimal determination coefficient, the corresponding maximum cutoff value is used as the maximum quantization cutoff value, thereby obtaining the minimum quantization cutoff value and the maximum quantization cutoff value.

[0062] The coefficient of determination is an indicator used to measure the goodness of fit of the maximum cutoff value. The larger the coefficient of determination, the better the goodness of fit.

[0063] In some embodiments, the expression for the coefficient of determination can be:

[0064]

[0065] Among them, R 2 The coefficient of determination is represented by x, which represents the activation value after activation. The quantization cutoff interval is represented by SSE, which represents the sum of squared residuals between the reference distribution and the quantization distribution. SST represents the total sum of squares between the reference distribution and the quantization distribution. quant_center represents the activation value, center represents the median of the interval, i represents the activation value index, count represents the number of activation values, and mean_center represents the mean of the original data.

[0066] In some embodiments, the expressions for the minimum and maximum quantization cutoff values ​​can be:

[0067] clip min =min(x);

[0068] clipmax =max(x);

[0069] SiLU(x') = x'·σ(x');

[0070]

[0071] Among them, clip min This represents the minimum value of the quantization cutoff, min represents the minimum value function, x represents the activation value after activation, and clip represents the minimum value. max σ represents the maximum value of the quantization cutoff, max represents the maximum value function, SiLU represents the SiLU function, x' represents the activation value of the input SiLU, and σ represents the Sigmoid function.

[0072] In some embodiments, the processor can further uniformly truncate activation values ​​outside the quantization cutoff range to obtain the corresponding minimum and maximum quantization cutoff values:

[0073]

[0074] S3: Based on the minimum and maximum quantization cutoff values, perform low-bit quantization on the candidate deployment model to obtain the target deployment model.

[0075] The target deployment model is a lightweight neural network model designed for deployment on edge computing platforms.

[0076] In some embodiments, the processor can generate quantization nodes by determining the quantization range of activation values ​​and weights, thereby obtaining the target deployment model.

[0077] In some embodiments, the processor can determine the quantization range of activation values ​​based on the selection of minimum and maximum values. For example, the use of SiLU as the activation function in YoloV8 results in an uneven distribution of activation values, with the minimum value generally around -0.28 and relatively densely packed. Directly selecting the minimum activation value as the minimum value of the quantization range can speed up quantization, reduce unnecessary calculations, and avoid precision loss due to truncation. The maximum activation value is determined by finding an activation value such that the R-squared of the truncated activation value and the original activation value is closest to 1; this truncation point (maximum value) is the upper bound of the quantization range. Low-bit quantization typically uses the truncation of the maximum and minimum values, or MSE or KL divergence for error analysis.

[0078] In some embodiments, the processor can use the coefficient of determination to calculate the quantized distribution information to obtain the corresponding regression fit.

[0079] Regression fit is data that reflects the degree of fit between the quantified distribution information and the reference distribution information.

[0080] In some embodiments, the expression for regression fit can be:

[0081]

[0082] Among them, R 2 ' represents the regression fit, and argmin represents the parameter that minimizes the function.

[0083] In some embodiments, the processor can perform performance analysis on each quantized target deployment model based on the regression fit to obtain the performance results of the target deployment model.

[0084] The target deployment model is a lightweight neural network model designed for deployment on edge computing platforms. For example, as shown in Table 1, the inference time per image for the YoloV8 target deployment model is approximately 20.8 ms, which can be accelerated to 4.2 ms after low-bit quantization alone, and further accelerated to 2.5 ms after pruning and low-bit quantization.

[0085] Table 1. Pruning Quantization Effect of the YOLOv8 Target Deployment Model

[0086] prune-rate P mAP50 speed YoloV8 0 0.825 0.901 19.0ms YoloV8-int8 0 0.817 0.897 4.2ms YoloV8-prune 0.5 0.816 0.835 4.5ms YoloV8-prune-int8 0.5 0.803 0.822 2.5ms

[0087] In some embodiments, the processor can process the input data as a 2048-bin FP32 format histogram H, where each bin represents the frequency of an interval. Each value of the data falls within a certain interval, and the frequency of that interval is quantized using candidate deployment models stored in that bin to obtain the target deployment model. For example, the processor can start from the 128th interval, extracting data from bin 0 to bin i as the current reference distribution P; calculate the sum of data in H from the i-th to the 2047th number, i.e., count the frequency after the cutoff point i. This part of the data is considered "outliers"; add c outlier counts to the last bin of the reference distribution P: P[i-1]; divide each value in the reference distribution P by the sum of P, making the sum of all probabilities equal to 1; and quantize the data selected from H[0:i] into 128 levels. This step discretizes the data using quantization techniques, typically mapping the data to a discrete range of levels; expands the quantized distribution C to the same number of bins as the reference distribution P; divides each value in C by the sum of C's values, making C an efficient probability distribution; calculates the residual sum of squares (SSE) between the reference distribution P and the quantized distribution C; calculates the total sum of squares (SST), which is the sum of squared differences between each value in the reference distribution P and its mean; and calculates R0. 2 (Coefficient of determination) measures the goodness of fit between the quantized distribution C and the reference distribution P, and is used to obtain the target deployment model.

[0088] Input task data refers to the processing task data of the edge computing platform. For example, input task data may include detection, segmentation, pose estimation, tracking, and classification.

[0089] Task processing results are the processing output data corresponding to the input task. For example, task processing results may include detection results, segmentation results, pose estimation results, tracking results, and classification results.

[0090] In some embodiments, the processor can deploy the target deployment model to the Jetson Nano edge computing platform, thereby reducing memory footprint and increasing inference speed. For example, the processor can import ONNX into Jetson Nano, perform 8-bit quantization using TeasorRT's trtexec tool, generate an engine file, and use the engine model for inference and accuracy testing to obtain the performance of the target deployment model.

[0091] In some embodiments of this specification, a model quantization method for edge computing platforms is provided. (1) A structured pruning method is adopted to gradually eliminate redundant parameters, thereby significantly reducing the model size while maintaining model accuracy as much as possible, thus reducing the number of calculations in the model inference process. Then, low-bit quantization is performed to convert floating-point operations into fixed-point operations, reducing computational complexity. The model is simultaneously compressed losslessly in terms of structure and data format, and then deployed to the edge computing platform, thereby reducing memory usage and increasing inference speed. (2) It has significant improvements in efficiency, adaptability, and real-time performance, achieving a good balance between model depth and performance, and improving the convenience of model deployment. The pruning-quantization joint optimization forms a complete process, reducing the need for manual parameter tuning. It supports one-click conversion of mainstream edge frameworks (TensorRT).

Claims

1. A model quantization method for edge computing platforms, characterized in that, include: S1: The model of the edge computing platform is iteratively pruned step by step to obtain candidate deployment models; wherein, the candidate deployment models are used to process the image dataset to obtain the activation values ​​corresponding to each layer, and the image dataset includes detection, segmentation, pose estimation, tracking and classification image data; S2: Calculate the activation value using an activation function to obtain the minimum and maximum quantization cutoff values; analyze the activation value corresponding to each layer to obtain the range of activation values ​​for each layer; Based on the range of activation values ​​for each layer, the minimum value of the activation value is fixed as the minimum value of the quantization cutoff. The maximum value of the activation value is traversed, and the distribution of the determination coefficients of each maximum cutoff value is obtained through calculation. The expression for the coefficient of determination is: ； ； ； in, The coefficient of determination is represented by the coefficient of determination. This represents the activation value after activation. This represents the sum of squared residuals between the reference distribution and the quantization distribution. This represents the total sum of squares between the reference distribution and the quantization distribution. This represents the activation value. This represents the midpoint of a range of values. Indicates the activation value sequence number. This indicates the quantity corresponding to the activation value. This represents the mean of the original data; Based on the distribution of the determination coefficients, the maximum value among them is selected as the optimal determination coefficient; Based on the optimal determination coefficient, the corresponding maximum truncation value is taken as the maximum quantization truncation value, and the minimum and maximum quantization truncation values ​​are obtained. S3: Based on the minimum and maximum values ​​of the quantization cutoff, perform low-bit quantization on the candidate deployment model to obtain the target deployment model, thus completing the model quantization of the edge computing platform; wherein, the target deployment model is used to process the input task data of the edge computing platform to obtain the task processing result.

2. The model quantization method for edge computing platforms according to claim 1, characterized in that, S1 includes: Based on the target pruning rate, the model of the edge computing platform is iteratively pruned step by step, resulting in the corresponding pruning accuracy loss: ； in, This indicates a loss of pruning precision. Indicates the target pruning rate. Indicates the number of iterations; When the pruning accuracy loss reaches a threshold, the iteration stops, and a candidate deployment model is obtained.

3. The model quantization method for edge computing platforms according to claim 1, characterized in that, The expressions for the minimum and maximum quantization cutoff values ​​are: ； ； ； ； in, This indicates the minimum value of the quantization cutoff. This represents the function that takes the minimum value. This represents the activation value after activation. This indicates the maximum value of the quantization cutoff. This represents the function that takes the maximum value. express function, This represents the activation value of the input SiLU. This represents the Sigmoid function.

4. The model quantization method for edge computing platforms according to claim 1, characterized in that, The minimum and maximum quantization cutoff values ​​include: further uniformly truncating activation values ​​outside the quantization cutoff range to obtain the corresponding minimum and maximum quantization cutoff values. 。

Images