Post-training quantization and fine-tuning method for point cloud object detection network based on knowledge distillation
Through the quantization and fine-tuning method based on knowledge distillation, the quantization error problem of Transformer point cloud object detection network during edge devices is solved, and efficient 8-bit quantization and accuracy maintenance are achieved, which is suitable for point cloud object detection of unmanned driving equipment.
Patent Information
- Application Number
- CN202411443602.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2044-10-16
AI Technical Summary
When the existing point cloud target detection network is deployed on edge devices with limited resources, the performance after quantization is degraded, especially the Transformer-based network has large quantization errors during 8-bit quantization, and ordinary knowledge distillation methods cannot effectively maintain the model. Accuracy.
A post-training quantization and fine-tuning method for point cloud object detection network based on knowledge distillation is proposed. By dividing the network into multiple blocks, using full-precision teacher network and quantitative student network for fine-tuning block by block, and the quantization parameters are updated through the distillation loss function. , combine sparsity processing and adaptive quantization adjustment to improve quantization accuracy.
The 8-bit quantization of the Transformer point cloud target detection network with a small amount of calibration data is realized, which maintains high accuracy, reduces quantization errors, and improves the detection effect of point cloud target detection. It is suitable for unmanned driving equipment.
Smart Images
Figure CN118966274B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of point cloud target detection, and in particular to a post-training quantization and fine-tuning method for a point cloud target detection network based on knowledge distillation. Background Art
[0002] Lidar-based 3D object detection networks have received widespread attention in the field of autonomous driving. It is very important to quickly and accurately perceive objects in the surrounding environment. In particular, some advanced methods use Transformer as a backbone or part of the detection head, such as DSVT and TransFusion. These models fully utilize the Transformer's ability to obtain features and achieve good detection results.
[0003] However, when it is deployed on edge devices with limited resources, the huge memory and computational requirements of the network become the main factors hindering deployment. Quantization is an effective solution that achieves model compression by reducing the number of bits representing activations and weights. Compared with quantization-aware training (QAT) that requires access to all labeled data and a large amount of computing resources, post-training quantization (PTQ) only requires a small amount of labeled data sets as calibration sets and does not require retraining the network, thereby shortening the quantization process and being suitable for rapid network deployment.
[0004] Existing PTQ methods achieve 8-bit or even 4-bit quantization by selecting appropriate quantization parameters, which are obtained through a small amount of calibration data. Some traditional methods directly use the calibration data to forward propagate to obtain the scale factor. On this basis, some methods use local metrics or global metric functions (such as MSE, cosine) to search for the best scale factor. In addition, some methods (such as AdaRound) add a type of quantization parameter called rounding value to adjust the rounding of weights.
[0005] Regarding the quantization of 3D object detection, QD-BEV achieves smaller size and faster speed than the original BevFormer on multi-camera 3D detection tasks through QAT and distillation methods. BiPointNet achieves network binarization by introducing entropy maximization aggregation (EMA) and layer-by-layer scale recovery (LSR) to effectively restore feature representation capabilities. BSC achieves network binarization by using shifted sparse convolutions and differentiable search strategies. Lidar-PTQ achieves 8-bit quantization of point cloud-based object detection networks through sparsity calibration methods and task-guided global positive losses. This method is the first to propose a quantization method for large-scale 3D object detection methods based on outdoor LiDAR in autonomous driving.
[0006] However, even if it is only quantized to 8 bits, directly applying these methods to the Transformer-based 3D point cloud object detection network will lead to a significant performance degradation. Compared with the point cloud network based on sparse convolution or 2D convolution, the quantization of the Transformer-based point cloud network will lead to a larger quantization error. The reasons are summarized as follows:
[0007] 1. Due to the sparsity of point clouds, there will be a large number of zero pixels, which will bring more significant feature map differences compared to dense RBG tasks.
[0008] 2. Due to the differences in the Transformer-based point cloud network structure, some common methods that use metric functions to search for optimal quantization parameters will lead to a decrease in model performance if used directly.
[0009] 3. The detection effects of target detection on different categories of the same dataset will vary greatly. At the same time, when using the structure of ordinary knowledge distillation to quantize the network, the teacher network will transfer inappropriate knowledge to the student network. Summary of the invention
[0010] Based on the technical problems existing in the background technology, the present invention proposes a post-training quantization and fine-tuning method for a point cloud target detection network based on knowledge distillation, which quantizes the target detection network while maintaining the accuracy of the network as much as possible, thereby achieving efficient reasoning of the model and ultimately improving the accuracy of the quantized target detection network.
[0011] The post-training quantization and fine-tuning method of the point cloud target detection network based on knowledge distillation proposed in the present invention is used to fine-tune the quantization parameters of the target detection network after initial quantization to obtain a fine-tuned target detection network, wherein the target detection network after initial quantization includes a student network and a teacher network, and the student network stores the initialized quantization parameters;
[0012] Deploying the fine-tuned target detection network in an unmanned driving device, so that the unmanned driving device performs point cloud target detection through the fine-tuned target detection network;
[0013] The fine-tuning process of the target detection network after the initial quantization is as follows:
[0014] S1. Build a fine-tuning dataset based on the point cloud dataset used by the full-precision network;
[0015] S2, dividing the teacher network and the student network into multiple blocks of equal size and fine-tuning the quantization parameters in each block in turn to fine-tune the student network;
[0016] S3, in fine-tuning Before the block, the teacher network and the student network The blocks before the first block are set to the quantized state and the quantized parameters of the student network after fine-tuning are passed to the teacher network. The blocks after the first block keep full precision, and the first The first block of the student network is set to full precision. The blocks are set to quantized state. , is the total number of blocks in the teacher network or the student network;
[0017] S4. Input a point cloud data into the teacher network and the student network at the same time and perform forward propagation. The output of the block and the teacher network The output comparison of each block is recorded as the first distillation loss , the overall output of the teacher network is compared with the overall output of the student network as the second distillation loss ;
[0018] S5, based on the first distillation loss and second distillation losses Construct a loss function to Quantization parameters of activations in blocks and the rounded value of the weight Perform back propagation update, and keep other parameters unchanged;
[0019] S6. Repeat steps S4 to S5 until the student network is completed. Fine-tune each block;
[0020] S7. Order , repeat steps S3 to S5 until fine-tuning of all blocks in the student network is completed.
[0021] Furthermore, the second distillation loss Comparison of dense heatmaps including student network and teacher network , Comparison of the final output heat map of the student network and the teacher network And the comparison of the regression box splicing matrix of the student network and the teacher network ;
[0022] The loss function in step S5 Specifically:
[0023]
[0024] in, are all pre-given hyperparameters. is the loss of the quantization parameter used to optimize the student network weights.
[0025] Further, in step S4, the overall output of the teacher network is compared with the overall output of the student network and recorded as the second distillation loss Specifically:
[0026] According to the heat map output by the teacher network, the front The teacher network transmits the generated BEV feature index to the student network, and the BEV feature map is a feature map from a bird's-eye view perspective;
[0027] The vector corresponding to the BEV feature index in the BEV feature map is used as Q, and the original BEV feature map is used as K and V, which are input into the transformer decoders of the teacher network and the student network respectively. After passing through the feedforward network, the category matrix and the regression box concatenation matrix containing N positions are output respectively.
[0028] Furthermore, after the teacher network and the student network obtain the category matrix and the regression box concatenation matrix, it is specifically:
[0029] Calculate the maximum value max_cls of each row of the teacher network;
[0030] Multiply the row dimension of the regression box concatenation matrix of the teacher network and the student network by the maximum value max_cls;
[0031] Multiply the dimension of the columns of the category matrices of the teacher network and the student network by the average accuracy cls_weight, where cls_weight is the average accuracy of each category of the teacher network on the entire training dataset.
[0032] The advantages of the post-training quantization and fine-tuning method of the point cloud target detection network based on knowledge distillation provided by the present invention are: Spercentile does not require complex hyperparameters and calibration processes, and compared with the baseline (benchmark quantization method), this embodiment has higher accuracy and better effect; the PTQ framework of this embodiment can be well applied to the Transformer-based 3D point cloud target detection network, and proposes additional adjustment schemes for the quantization of the point cloud network with TransFusionHead as the detection head; overall, we only use a small amount of calibration data and can quantize the weights and activations of this type of 3D point cloud target detection network to 8 bits, not all data sets, and there is only a small loss of accuracy while quantizing the network, which realizes the fast reasoning of the point cloud 3D target detection network, thereby improving the detection effect of point cloud target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a schematic diagram of the structure flow of the present invention;
[0034] Figure 2 Schematic diagram of the structure of passing the BEV feature index generated by the teacher network to the student network;
[0035] Figure 3 This is a schematic diagram of the structure of the Modify module. DETAILED DESCRIPTION
[0036] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0037] like Figures 1 to 3 As shown, the post-training quantization and fine-tuning method of the point cloud target detection network based on knowledge distillation proposed in the present invention is used to fine-tune the quantization parameters of the target detection network after initial quantization to obtain a fine-tuned target detection network, wherein the target detection network after initial quantization includes a student network and a teacher network, and the student network stores the initialized quantization parameters;
[0038] The fine-tuned target detection network is deployed in an unmanned driving device, so that the unmanned driving device performs point cloud target detection through the fine-tuned target detection network.
[0039] This embodiment addresses the problem that existing network quantization methods cannot be well applied to Transformer-based point cloud 3D object detection networks. This embodiment aims to quantify the object detection network while maintaining the accuracy of the network as much as possible, thereby achieving efficient reasoning of the network, which is described in detail below.
[0040] This embodiment is aimed at Transformer-based point cloud object detection networks, such as Dynamic Sparse Voxel Transformer with Spin Sets (DSVT) and TransFusion-L, which is a robust 3D object detector that fuses radar cameras through transformers (Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers). A PTQ quantization method based on the knowledge distillation framework is proposed for the characteristics of these point cloud networks. PTQ is a general term for a class of quantization methods, which means post-training quantization. Specifically, the following parts are proposed:
[0041] A. Spercentile: Network quantization initialization method based on sparsity
[0042] A simple and effective quantization parameter initialization method Spercentile is proposed to address the sparsity of point clouds:
[0043] Compared with 2D detection or classification networks with regular input, point clouds are very sparse. For example, when the point cloud of the current mainstream outdoor 3D target detection dataset nuscenes (a commonly used dataset in the field of autonomous driving) is projected onto the corresponding RGB image, only about 5% of the pixels have corresponding point clouds. The impact of this is that there are more zero pixels in the feature map of the point cloud network than in the 2D task, so some 2D quantization methods will have a large loss of accuracy when directly used on 3D point clouds.
[0044] Based on this, this embodiment proposes a simple and effective network quantization parameter initialization method based on sparsity, Spercentile, which is an improvement on the percentile method and adds a sparse pre-processing process. The percentile method is used to calculate the specified percentile of array elements on a specified dimension, and the return value is a scalar or array. Specifically, in the calibration phase, the number of 0 values in the feature map is counted in advance, and a certain proportion is added. The 0 value of is removed from the tensor and then participates in the percentile operation, saving only the and Percentile activation values for each layer, where , After the experiment of this embodiment, on DSVT and TransFusion-L, The effect is best when
[0045] B. PTQ framework (post-training quantization framework)
[0046] A knowledge distillation-based PTQ framework for transformer point cloud networks
[0047] Although Spercentile can effectively improve the quantization accuracy of the lidar detector, there is still a large gap compared to the full precision. Therefore, this embodiment proposes a PTQ framework based on knowledge distillation. As shown in Figure 1, this framework is used to fine-tune the quantization parameters of the target detection network after the quantization parameters are initialized. Compared with the previous method, this framework has the following three improvements:
[0048] b1) A PTQ quantification framework and process based on knowledge distillation
[0049] Before fine-tuning using this framework, the quantized initialization parameters obtained by the Spercentile method (i.e., the network quantization initialization method based on sparsity) have been stored in the student network, but the student network is not quantized at this time. In this framework, the teacher network plays the role of the network before quantization, and the student network plays the role of the network after quantization.
[0050] The fine-tuning process of the target detection network in this embodiment is as follows:
[0051] S1. Build a fine-tuning dataset based on the point cloud data used by the full-precision network.
[0052] S2, dividing the teacher network and the student network into multiple blocks of equal size and fine-tuning the quantization parameters in each block in turn to fine-tune the student network;
[0053] It can be understood that the number of blocks divided in the teacher network and the student network is consistent. In this embodiment, N blocks are set, and parameters of each block in the student network are fine-tuned.
[0054] S3, in fine-tuning Before the block, the teacher network and the student network The blocks before the first block are set to the quantized state and the quantized parameters of the student network after fine-tuning are passed to the teacher network. The blocks after the first block keep full precision, and the first The first block of the student network is set to full precision. The blocks are set to quantized state. , is the total number of blocks in the teacher network or the student network;
[0055] As shown in Figure 1, Block indicates that the current block is a full-precision block, and QBlock indicates that the current block is a quantized block.
[0056] S4. Input a point cloud data into the teacher network and the student network at the same time and perform forward propagation. After one forward propagation, the student network The output of the block and the teacher network The output comparison of each block is recorded as the first distillation loss , the overall output of the teacher network is compared with the overall output of the student network as the second distillation loss ;
[0057] Among them, the second distillation loss It consists of three parts: Comparison of dense heat maps of student network and teacher network , Comparison of the final output heat map of the student network and the teacher network And the comparison of the regression box splicing matrix of the student network and the teacher network .
[0058] S5. Feature map based and feature map Construct a loss function to Quantization parameters of activations in blocks and the rounded value of the weight Perform back propagation update, and keep other parameters unchanged;
[0059] Loss Function Specifically:
[0060]
[0061] in, are all pre-given hyperparameters. is the loss of the quantization parameter used to optimize the student network weights.
[0062] against It can be understood that existing quantization methods (such as quantization methods: Adaptive Rounding (AdaRound), Breaking the Limits of Post-training Quantization by Block Reconstruction (BRECQ) and Random Drop Quantization (QDrop) have made significant progress on PTQ (post-training quantization). These existing quantization methods optimize a continuous variable called a rounding value for each weight value to be quantized to determine whether to round up or round down the weight value during the quantization process. This method is orthogonal to the quantization and fine-tuning method of this embodiment. In order to obtain better accuracy, this embodiment also uses it. The loss is recorded as .
[0063] S6. Repeat steps S4 to S5 until the student network is completed. Fine-tune each block.
[0064] S7. Order , repeat steps S3 to S5 until fine-tuning of all blocks in the student network is completed.
[0065] b2) Make adaptive quantitative adjustments to the detection head of the 3D point cloud
[0066] Since this embodiment is aimed at the Transformer-based 3D point cloud target detection network, TransFusionHead, as a general transformer-based detection head, has been widely used in some classic networks such as DSVT, TransFusion-L and other point cloud networks, which greatly improves the model's ability to capture features. However, when the process in b1) is directly used to fine-tune the network, the effect is not ideal, and even the accuracy is reduced. Therefore, it is necessary to make adaptive quantitative adjustments to the detection head.
[0067] TransFusionHead is a TransFormer-based detection head proposed in the TransFusion paper, which is a powerful 3D object detector that fuses radar cameras through a transformer. Specifically, given a BEV characteristic diagram of dimension , is the width and height of the BEV feature map, from which a class-related heatmap is first predicted , K is the number of categories, and the most likely category can be obtained according to the heatmap. The most likely target BEV feature index, and then according to these BEV feature indexes Find the corresponding vector as Q in As K and V, they are input into the transformer decoder of the teacher network and the student network respectively, and after passing through the feed-forward neural network (FFN), the final output is obtained, which contains the category matrix and regression box concatenation matrix of N positions.
[0068] However, due to the existence of quantization error, the student network and the teacher network have different N BEV feature indexes that are most likely to be targeted. This has no effect on normal network reasoning. Since this embodiment fine-tunes the quantization parameters, If the compared category matrix and regression box concatenation matrix are not obtained by forward propagation of the features of the same N positions through the target detection network, the error will certainly be large, because the second distillation loss at this time The features being compared may not be of the same category or corresponding position, so back propagation cannot make the loss function convergence.
[0069] Therefore, this embodiment uses a simple and effective method to solve this problem. After the teacher network obtains the BEV feature index of the most likely target corresponding to the current input point cloud data under the BEV (bird's eye view) perspective, the BEV feature index generated by the teacher network is directly passed to the student network, such as Figure 2 As shown. The second distillation loss The comparison is between the category matrix and the regression box concatenation matrix generated by the features at the same position. Since the positions where the BEV feature indexes of the teacher network and the student network are different are likely to be untargeted or will be removed by subsequent operations, this operation will hardly affect the performance of the student network, but for the loss function that requires backpropagation For BEV, this difference in feature indexing will produce an unreasonable loss function. , therefore, the method of this embodiment is effective and reasonable.
[0070] b3) Make adaptive quantitative adjustments to the output of the 3D point cloud
[0071] Since the number of samples of different categories in the training data set used by the 3D object detection network varies greatly, the detection accuracy of the trained object detection network for different categories varies greatly. For example, the average precision (AP) of DSVT for cars in the nuscenes data set is 0.8816, but the AP for engineering vehicles is only 0.2525. Therefore, this embodiment should consider the gap between categories when transferring knowledge to the student network, so that the student network can learn useful knowledge as much as possible. On the other hand, since the confidence of different prediction boxes is different in the detection task, this embodiment wants the student network to learn as much knowledge related to the target as possible rather than some background or prediction boxes that the teacher network believes have a very low probability of containing the target.
[0072] Therefore, this embodiment proposes a Modify module (modification module), such as Figure 3 As shown. Specifically, after obtaining the category matrix and regression box concatenation matrix detected by the teacher network and the student network: calculate the maximum value max_cls of each row of the teacher network; multiply the row dimension of the regression box concatenation matrix of the teacher network and the student network by max_cls; multiply the column dimension of the category matrix of the teacher network and the student network by cls_weight, where cls_weight is the average precision (AP) of each category of the teacher network on the entire training data set.
[0073] C. Other settings
[0074] Regarding the quantization of some nonlinear layers in DSVT, such as layer normalization (LayerNorm), activation function (Softmax), this embodiment follows the quantization method in FQ-ViT, and the GELU operator (activation function) is replaced by the easier-to-implement hardSigmoid (hardSigmoid is a type of activation function) in the paper "Searching for MobileNetV3" published by Google on arXiv. Overall, this embodiment quantizes the weights and activations of the target detection network to 8 bits, where MobileNetV3 is the network architecture proposed by Google on March 21, 2019, and Searching for MobileNetV3 is to complete MobileNetV3 using neural structure search (NAS).
[0075] Currently in the field of autonomous driving, the only quantization method for large-scale outdoor LiDAR-based 3D target detection is LiDAR-PTQ (post-training quantization for point cloud three-dimensional object detection). However, this is for CNN networks (convolutional neural networks). No one has yet provided a solution for the quantization of TransFormer-based 3D point cloud target detection networks.
[0076] Specifically, this embodiment proposes a simple and effective quantization parameter initialization method Spercentile for the sparsity of point clouds, and based on this, proposes a PTQ framework based on knowledge distillation suitable for transformer point cloud networks.
[0077] The advantages of this embodiment are: Spercentile does not require complex hyperparameters and calibration processes, and compared with the baseline model, this embodiment has higher accuracy and better results; the PTQ framework of this embodiment can be well applied to the Transformer-based 3D point cloud target detection network, and proposes additional adjustment schemes for the quantization of the point cloud network with TransFusionHead as the detection head; overall, we only use a small amount of calibration data and can quantize the weights and activations of this type of 3D point cloud target detection network to 8 bits, not all data sets, and there is only a small loss of accuracy while quantizing the network, which realizes the fast reasoning of the point cloud 3D target detection network, thereby improving the detection effect of point cloud target detection.
[0078] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A post-training quantization and fine-tuning method for point cloud object detection network based on knowledge distillation, characterized in that: Used to fine-tune the quantization parameters of the target detection network after initial quantization to obtain a fine-tuned target detection network, wherein the target detection network after initial quantization includes a student network and a teacher network, and the student network stores the initialized quantization parameters; Deploying the fine-tuned target detection network in an unmanned driving device, so that the unmanned driving device performs point cloud target detection through the fine-tuned target detection network; The fine-tuning process of the target detection network after the initial quantization is as follows: S1. Build a fine-tuning dataset based on the point cloud dataset used by the full-precision network; S2, dividing the teacher network and the student network into multiple blocks of equal size and fine-tuning the quantization parameters in each block in turn to fine-tune the student network; S3, in fine-tuning Before the block, the teacher network and the student network The blocks before the first block are set to the quantized state and the quantized parameters of the student network after fine-tuning are passed to the teacher network. The blocks after the first block keep full precision, and the first The first block of the student network is set to full precision. The blocks are set to quantized state. , is the total number of blocks in the teacher network or the student network; S4. Input a point cloud data into the teacher network and the student network at the same time and perform forward propagation. The output of the block and the teacher network The output comparison of each block is recorded as the first distillation loss , the overall output of the teacher network is compared with the overall output of the student network as the second distillation loss ; S5, based on the first distillation loss and second distillation losses Construct a loss function to Quantization parameters of activations in blocks and the rounded value of the weight Perform back propagation update, and keep other parameters unchanged; S6. Repeat steps S4 to S5 until the student network is completed. Fine-tune each block; S7. Order , repeat steps S3 to S5 until fine-tuning of all blocks in the student network is completed.
2. The method for post-training quantization and fine-tuning of a point cloud target detection network based on knowledge distillation according to claim 1, characterized in that: Second distillation loss Comparison of dense heatmaps including student network and teacher network , Comparison of the final output heat map of the student network and the teacher network And the comparison of the regression box splicing matrix of the student network and the teacher network ; The loss function in step S5 Specifically: in, are all pre-given hyperparameters. is the loss of the quantization parameter used to optimize the student network weights.
3. The method for post-training quantization and fine-tuning of a point cloud target detection network based on knowledge distillation according to claim 1, characterized in that: In step S4, the overall output of the teacher network is compared with the overall output of the student network and recorded as the second distillation loss Specifically: According to the heat map output by the teacher network, the front The teacher network transmits the generated BEV feature index to the student network, and the BEV feature map is a feature map from a bird's-eye view perspective; The vector corresponding to the BEV feature index in the BEV feature map is used as Q, and the original BEV feature map is used as K and V, which are input into the transformer decoders of the teacher network and the student network respectively. After passing through the feedforward network, the category matrix and the regression box concatenation matrix containing N positions are output respectively.
4. The method for post-training quantization and fine-tuning of a point cloud target detection network based on knowledge distillation according to claim 3 is characterized in that: After the teacher network and the student network obtain the category matrix and the regression frame concatenation matrix, the specific ones are: Calculate the maximum value max_cls of each row of the teacher network; Multiply the row dimension of the regression box concatenation matrix of the teacher network and the student network by the maximum value max_cls; Multiply the dimension of the columns of the category matrices of the teacher network and the student network by the average accuracy cls_weight, where cls_weight is the average accuracy of each category of the teacher network on the entire training dataset.