A method for accelerating 3D object detection using pruning and distillation

Abstract

This invention proposes a method for accelerating 3D object detection using pruning and distillation, relating to the field of computer vision technology. The method comprises five steps: constructing a well-trained multimodal expert model; building a module-aware genetic channel search pruning strategy; constructing a multimodal 3D student model based on the pruning strategy; constructing a joint loss distillation strategy between the teacher model and the student model; training the model; and performing 3D object detection based on the knowledge-distilled student model. This invention improves the model's structure and performance while significantly reducing the number of parameters and computational load while maintaining high detection accuracy. By reducing redundant parameters and computational load, the model's inference speed is greatly improved, making this method particularly suitable for embedded devices and other scenarios with limited computing resources, thereby promoting the widespread adoption of 3D object detection technology in practical applications.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for accelerating 3D object detection using pruning and distillation. Background Art

[0002] In the field of autonomous driving, 3D object detection is crucial for vehicle perception systems, especially for object recognition and obstacle avoidance in complex environments. Multimodal fusion perception methods based on cameras and LiDAR (LiDAR) effectively improve perception accuracy and robustness by utilizing visual information from cameras and spatial distance information from LiDAR. However, despite their high accuracy, multimodal fusion methods typically suffer from slow inference speeds, particularly in autonomous driving applications requiring real-time responses, posing significant challenges to deployment on embedded devices. This is primarily due to limitations in computing resources, leading to high deployment costs and hindering the widespread adoption of this technology.

[0003] In image classification, common model compression methods, such as pruning, quantization, and distillation, have been widely used to improve inference speed, reduce model storage footprint, and attempt to find a balance between accuracy and speed. Pruning methods reduce computation and improve inference speed by reducing redundant weights in the model and retaining the most important parts for the final decision. In 3D object detection tasks, although pruning can theoretically improve inference speed, due to the high accuracy requirements of this task, pruning methods often lead to significant accuracy loss, especially when combined with multimodal data, where key feature information may be lost after pruning.

[0004] Meanwhile, model distillation, as another compression technique, can effectively improve the performance of smaller models by transferring knowledge from large teacher models to smaller student models, reducing computational complexity while maintaining inference accuracy. However, in the field of 3D object detection, the pursuit is mostly for higher detection accuracy, and model compression, especially the combination of pruning and distillation, is not common, which limits the practical deployment of most models. Therefore, this invention proposes a method to accelerate 3D object detection using pruning and distillation to solve the problems existing in the prior art. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to propose a method for accelerating 3D object detection using pruning and distillation. This method, through pruning and distillation techniques, reduces model complexity and computational load while maintaining high detection accuracy and robustness, thus solving the problems existing in the prior art.

[0006] To achieve the objectives of this invention, the invention is implemented through the following technical solution: a method for accelerating 3D object detection using pruning and distillation, comprising the following steps:

[0007] Step 1: Construct a well-trained multimodal expert model

[0008] A well-trained multimodal 3D target detection model based on image and LiDAR fusion is obtained as the teacher model. The teacher model includes an image feature extraction network, a LiDAR point cloud feature extraction network, a multimodal BEV feature fusion module, a shared BEV feature extraction module, and a 3D target detection head. Then, a dataset is constructed and processed to obtain the processed dataset, which is then input into the teacher model for feature extraction to obtain feature maps.

[0009] Step 2: Construct a module-aware genetic pathway search and pruning strategy

[0010] A module-aware genetic channel search and pruning strategy is constructed hierarchically for each module of the teacher model. Based on the genetic algorithm, the channels of each module are encoded separately.

[0011] Step 3: Construct a multimodal 3D student model based on the pruning strategy.

[0012] A 3D target detection model fused from image and LiDAR point cloud is constructed as a student model. The optimal individual is selected in step two. Based on the selected optimal individual, i.e. the optimal channel selection of the module, the weight of the optimal channel is assigned to the channel of the corresponding module of the student model. The weights of other unselected channels are set to 0, which are channels to be pruned. The weights of the selected channels of each module are taken out in sequence and assigned to the student model. The process in step two is iterated to calculate the fitness value, and finally a student model with the optimal number of channels and structure is formed.

[0013] Step 4: Construct a joint loss distillation strategy between the teacher model and the student model, and train them.

[0014] Constructing a joint loss distillation strategy for the teacher model and the student model, the loss from knowledge distillation of the teacher model and the student model is denoted as the BEV feature loss L. bev And the predicted result loss L head Total loss L = L bev +L head Thus, different knowledge is transmitted to the student model through multiple types of joint loss, and then the student model is trained under supervision.

[0015] Step 5: Perform 3D target detection based on the student model after knowledge distillation.

[0016] Based on pruning and distillation strategies, and a trained student model, a 3D object detection task is performed, where the data input, image processing, and enhancement methods are the same as in step one.

[0017] A further improvement is that, in step one, the data processing includes image enhancement processing and point cloud data enhancement processing, both of which include cropping and rotation processing.

[0018] A further improvement is that, in step one, the image feature extraction network includes an image feature encoding module and an LSS-like view space transformation module.

[0019] A further improvement is made in the following: In step one, the lidar point cloud feature extraction network includes a point cloud voxel feature encoder and a BEV feature extraction module.

[0020] A further improvement lies in the following: In step two, the encoding method is as follows:

[0021] S1: For specific modules The convolutional channels are encoded using binary genotypes to generate population P. N One individual in the population is

[0022] S2: Define the number of genetic iterations as iters times, and then iterate through the initial population P. N Through the fitness function f fit Calculate fitness value V fit Based on the fitness value, the roulette wheel selection probability is calculated, and the roulette wheel selection probability method is used to select K individuals for the population.

[0023] S3: Perform crossover and mutation operations on the K individuals with the best fitness values ​​in the population using a genetic algorithm. Genetic crossover will generate M1 new individuals, and genetic mutation will generate M2 new individuals. Use the inherited K+M1+M2 individuals to update the population.

[0024] S4: Denote the population as Perform the next genetic iteration on the population until an individual in the population has reached the desired fitness value or the maximum number of iterations has been reached.

[0025] The further improvement lies in: the fitness function f fit Defined as:

[0026]

[0027] In the formula, ACT is the sigmoid activation function with an output range of 0 to 1, and cls_heatmap ori and cls_heatmap curThese are heatmaps of categories before and after pruning, N. cls This represents the number of categories for the object detection task.

[0028] A further improvement lies in the following: In step three, the specific steps for constructing the BEV feature loss are as follows:

[0029] BEV features at corresponding locations in the teacher and student models are sampled, including image BEV features F. cbev LiDAR point cloud BEV features F lbev Multimodal fusion of BEV features F fbev Using the real bounding box gt_bbox3d, generate a mask with the same size as the three BEV features mentioned above. mask Multiplying the BEV features by each BEV feature yields foreground-aware BEV features, which are then used to calculate the F-values ​​of the teacher and student models. cbev F lbev F fbev Feature loss, respectively L cbev 、L lbev 、L fbev The image BEV feature loss L cbev LiDAR BEV characteristics lbev and multimodal fusion of BEV features L fbev The sum of these features constitutes the total BEV characteristic loss L. bev .

[0030] A further improvement lies in the following: In step three, the specific method for constructing the prediction result loss is as follows:

[0031] Extract the prediction results P from the teacher model and the student model. T ,P S The student's model prediction P S Teacher model predicts P T Calculate the loss L pred The model predicts P for students. S Compared with the true result, GT loss L gt The prediction loss L from the teacher model pred The loss L compared to the true result gt The sum of these constitutes the total predicted loss L. head .

[0032] The beneficial effects of this invention are as follows:

[0033] (1) This invention designs a pruning and distillation framework for 3D object detection tasks, effectively combining two model compression techniques to achieve efficient model inference. Compared with the traditional method of using pruning or distillation alone, this invention can simultaneously achieve both accuracy and inference speed.

[0034] (2) This invention constructs an automatic channel search strategy based on a genetic algorithm, which can prune the convolutional channels in the teacher model module by module. The genetic algorithm can adaptively select the optimal pruning strategy, automatically discover the most suitable channel combination for a specific task, maximize the accuracy of the pruned model, and effectively reduce the amount of computation.

[0035] (3) This invention uses knowledge distillation to transfer the rich knowledge of the teacher model to the student model, and uses the generated foreground mask to focus on learning local important features, helping the student model recover the loss caused by pruning, while learning deeper features.

[0036] (4) This invention improves the structure and performance of the model, and significantly reduces the number of parameters and computational load while maintaining high detection accuracy. By reducing redundant parameters and computational load, the inference speed of the model is greatly improved, making this method particularly suitable for embedded devices and other scenarios with limited computing resources, thereby promoting the popularization of 3D target detection technology in practical applications. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the steps of the present invention.

[0038] Figure 2 This is a schematic diagram of the model structure and distillation of the present invention.

[0039] Figure 3 This is a schematic diagram of the teacher model structure of the present invention.

[0040] Figure 4 This is a schematic diagram of the pruning strategy process of the present invention. Detailed Implementation

[0041] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0042] As can be seen from the background technology, in the field of 3D target detection, most people pursue higher detection accuracy, and have limited consideration for the actual deployment and application of most models, especially the model compression combining pruning and distillation is not common.

[0043] Current popular 3D object detection model compression techniques mostly involve single-modal knowledge distillation, where a high-performance pure laser or multimodal teacher model is used to distill a low-performance pure vision student model. Although many works have demonstrated the effectiveness of distillation methods, the actual acceleration effect is still unsatisfactory and cannot meet the real-time operation requirements of embedded devices.

[0044] To address the above problems, this invention provides a method for accelerating 3D object detection using pruning and distillation, as detailed below:

[0045] according to Figure 1 As shown, this embodiment proposes a method for accelerating 3D object detection using pruning and distillation, including the following steps:

[0046] Step 1: Construct a well-trained multimodal expert model

[0047] A well-trained multimodal 3D target detection model based on image and LiDAR fusion is obtained as the teacher model, and then a student model is obtained. In this embodiment, the student model is isomorphic to the teacher model, with only differences in the number of convolution channels and parameters. Then, a dataset is constructed and data processing is performed. In this embodiment, the NuScenes large public dataset for autonomous driving is used, from which six 1600×900×3 surround-view RGB images and one 360-degree LiDAR point cloud image are loaded. The data processing includes image enhancement processing and point cloud data enhancement processing. Both image enhancement processing and point cloud data enhancement processing include cropping and rotation processing. Specifically, the image enhancement cropping scales the 1600×900×3 RGB image to 704×256×3, while the point cloud data enhancement cropping extracts only the point cloud data in the range of -54.0, -54.0, -5.0, 54.0, 54.0, 3.0, where the ranges in the X and Y axes are both -54.0, 54.0, and the range in the Z axis is -5.0, 3.0. After that, the processed data is fed into the teacher model to begin feature extraction.

[0048] For teacher models such as Figure 2 and Figure 3 As shown, it includes an image feature extraction network, a LiDAR point cloud feature extraction network, a multimodal BEV feature fusion module, a shared BEV feature extraction module, and a 3D target detection head. The image feature extraction network includes an image feature encoding module and an LSS-like view space transformation module (vtransform), and the LiDAR point cloud feature extraction network includes a point cloud voxel feature encoder and a BEV feature extraction module.

[0049] It is important to emphasize that the teacher model in this embodiment of the invention is a well-trained multimodal 3D target detection model based on image and LiDAR fusion. The overall structure and feature extraction network of the teacher model are similar to the BEVFusion model. The teacher model extracts image features through an image feature encoding module, then converts 2D image features into 3D geometric space features through an LSS-like view space transformation module, and then obtains image BEV features through BEV pooling. The teacher model obtains voxel features through a point cloud voxel feature encoder, and the BEV feature extraction module converts the voxel features into point cloud BEV features. The multimodal BEV feature fusion module of the teacher model concatenates the image BEV features and point cloud BEV features by dimension, and then fuses them through convolution operations to form multimodal BEV features. The BEV features are fed into a shared BEV feature extraction module to extract deeper BEV features, and these features are then fed into the 3D target detection head. The output of the 3D target detection head includes a BEV feature classification heatmap, target prediction classification results, and target prediction regression boxes.

[0050] Step 2: Construct a module-aware genetic pathway search and pruning strategy

[0051] A module-aware genetic channel search and pruning strategy is constructed hierarchically for each module of the teacher model. Based on the genetic algorithm, the channels of each module are encoded separately.

[0052] According to the genetic algorithm, the channels of each module need to be encoded separately. Obviously, encoding all modules simultaneously is unreasonable, as it would lead to a huge search space, make the search difficult, and waste GPU memory and other resources. Therefore, encoding channels by module and level is particularly important. Following the traditional iterative method of the genetic algorithm, the convolutional channels of each module in the teacher model are encoded using binary genotypes. The specific encoding method is as follows:

[0053] S1: For specific modules 1≤i≤N model ,right The convolutional channels are encoded using binary genotypes to generate population P. N One individual in the population is in, Defined as:

[0054]

[0055] It is a 0-1 binary code, representing the code of the j-th channel of the i-th individual in the population. This means the channel will be cut off. This means the channel will be reserved;

[0056] Population P N Defined as:

[0057]

[0058] S2: Define the number of genetic iterations as iters times, and then iterate through the initial population P. N Through the fitness function f fit Calculate fitness value V fit Based on the fitness values, the roulette wheel selection probability is calculated. Using the roulette wheel selection probability method, K individuals are selected from the population, where the fitness function f... fit Defined as:

[0059]

[0060] In the formula, ACT is the sigmoid activation function with an output range of 0 to 1, and cls_heatmap ori and cls_heatmap cur These are heatmaps of categories before and after pruning, N. cls The number of categories for the object detection task;

[0061] The formula for calculating the probability of roulette is defined as follows:

[0062]

[0063] The formula for genetic crossover is defined as follows:

[0064]

[0065] Among them, mask cross A random vector mask consisting of 0s and 1s, used to randomly select individuals in the population. and The location of crisscross inheritance to generate new individuals.

[0066] The formula for genetic variation manipulation is defined as follows:

[0067]

[0068] RandomMut() is used for random toggling. The function of the encoded bits, where m_p is the mutation flip probability;

[0069] S3: Perform crossover and mutation operations on the K individuals with the best fitness values ​​using a genetic algorithm. Crossover generates M1 new individuals, and mutation generates M2 new individuals, where K + M1 + M2 = N. gen Using the inherited K+M1+M2 populations to update the population, the population is denoted as...

[0070] S4: Denote the population as Perform the next genetic iteration on the population until an individual in the population has reached the desired fitness value or the maximum number of iterations has been reached.

[0071] Step 3: Construct a multimodal 3D student model based on the pruning strategy.

[0072] A 3D target detection model fused from image and LiDAR point cloud is constructed as the student model. The optimal individual is selected in step two. Based on the selected optimal individual, i.e., the optimal channel selection of the module, a binary channel mask is generated, which represents the population individual in step two. The formula, during the training phase, will... The weights of the selected channels are assigned to the corresponding channels of the student model. The weights of other unselected channels are set to 0 to test the fitness value. The weights of the selected channels of each module are taken out in turn and assigned to the student model. The process in step two is iterated to calculate the fitness value and finally form a student model with the optimal number of channels and structure.

[0073] Step 4: Construct a joint loss distillation strategy between the teacher model and the student model, and train them.

[0074] Constructing a joint loss distillation strategy for the teacher model and the student model, the loss from knowledge distillation of the teacher model and the student model is denoted as the BEV feature loss L. bev And the predicted result loss L head Total loss L = L bev +L head Thus, different knowledge is conveyed to the student model through multiple types of joint loss;

[0075] The specific steps for constructing the BEV feature loss are as follows:

[0076] BEV features at corresponding locations in the teacher and student models are sampled, including image BEV features F. chev LiDAR point cloud BEV features F lbev Multimodal fusion of BEV features F fbev Using the real bounding box gt_bbox3d, generate a mask with the same size as the three BEV features mentioned above. mask Multiplying the BEV features by each BEV feature yields foreground-aware BEV features, which are then used to calculate the F-values ​​of the teacher and student models. cbev F lbev F fbev Feature loss, respectively L cbev 、L lbev 、L fbevThe image BEV feature loss L cbev LiDAR BEV characteristics lbev and multimodal fusion of BEV features L fbev The sum of these features constitutes the total BEV characteristic loss L. bev L bev =L cbev +L lbev +L fbev ;

[0077] Determine the BEV feature loss L bev for:

[0078]

[0079] in,

[0080] C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0081] This is a foreground 0-1 mask matrix formed by mapping the ground truth bounding box gt_bbox3d, used to filter out feature foreground objects;

[0082]

[0083] and The image BEV features are for the teacher and student models, respectively;

[0084] and These are the BEV features of the point cloud branches for the teacher and student models, respectively.

[0085] and These are the multimodal fusion BEV features of the teacher and student models, respectively. The teacher image BEV feature map BEV feature maps of student model images Forecast loss L cbev ;

[0086] BEV feature map of the teacher point cloud BEV feature map of student model point cloud Forecast loss L lbev ;

[0087] The teacher's multimodal BEV feature map Student model multimodal BEV feature map Forecast loss L fbev ;

[0088] In particular, this loss calculation method is similar to the MSE loss function;

[0089] Next, a prediction loss is constructed, and the prediction results P of the teacher model and the student model are extracted. T ,P S The student's model prediction P S Teacher model predicts P T Calculate the loss L pred The model predicts P for students. S Compared with the true result, GT loss L gt The prediction loss L from the teacher model pred The loss L compared to the true result gt The sum of these constitutes the total predicted loss L. head L head =βL gt +(1-β)L pred ;

[0090] Determine the total prediction loss L head for:

[0091]

[0092] in,

[0093] β is the hyperparameter for balancing the detection loss, and L is the hyperparameter for balancing the prediction loss of the student model itself. gt and distillation loss L pred ;

[0094] x∈{cls,bbox,heatmap} represents the student model prediction result, the teacher model prediction result, and the ground truth annotation information, respectively.

[0095] FL, L1, and GFL are respectively the FocalLoss function for correcting sample imbalance, the L1 loss function for penalized regression, and the Gaussian FocalLoss function based on the center range.

[0096]

[0097] The prediction results of the student model and real labeling information The loss;

[0098]

[0099] The prediction results of the student model Teacher model prediction results The loss.

[0100] Step 5: Perform 3D target detection based on the student model after knowledge distillation.

[0101] Based on pruning and distillation strategies, and a trained student model, a 3D object detection task is performed, where the data input, image processing, and enhancement methods are the same as in step one.

[0102] By employing a module-aware genetic pathway search pruning strategy, the number of parameters and computational cost of the multimodal model are reduced, resulting in a more compact student model. Combined with joint loss knowledge distillation, knowledge from the teacher model is transferred to the student model, improving the robustness of the student model, mitigating the accuracy loss caused by pruning, and ultimately enhancing the detection accuracy of the student model.

[0103] The training and processing modules and processes described above are performed on NuScenes, a large public dataset for autonomous driving. The performance of the pruned student models is evaluated using the NuScenes validation set, including metrics such as 3D detection accuracy, number of parameters, computational cost, and inference time.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its framework and scope of application, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for accelerating 3D object detection using pruning and distillation, characterized in that: Includes the following steps: Step 1: Construct a well-trained multimodal expert model A well-trained multimodal 3D target detection model based on image and LiDAR fusion is obtained as the teacher model. The teacher model includes an image feature extraction network, a LiDAR point cloud feature extraction network, a multimodal BEV feature fusion module, a shared BEV feature extraction module, and a 3D target detection head. Then, a dataset is constructed and processed to obtain the processed dataset, which is then input into the teacher model for feature extraction to obtain feature maps. Step 2: Construct a module-aware genetic pathway search and pruning strategy A module-aware genetic channel search and pruning strategy is constructed hierarchically for each module of the teacher model. Based on the genetic algorithm, the channels of each module are encoded separately. Step 3: Construct a multimodal 3D student model based on the pruning strategy. A 3D target detection model fused from image and LiDAR point cloud is constructed as a student model. In step two, the optimal individual is selected. Based on the selected optimal individual, i.e., the optimal channel selection of the module, the weights of the optimal channels are assigned to the corresponding channels of the student model. The weights of other unselected channels are set to 0, representing channels to be pruned. The weights of the selected channels from each module are sequentially extracted and assigned to the student model. The process in step two is iterated, and the fitness value is calculated. Finally, a student model with the optimal number of channels and structure is formed. The specific steps for constructing the BEV feature loss in step three are as follows: BEV features at corresponding locations in the teacher and student models are sampled, including image BEV features. LiDAR point cloud BEV features Multimodal fusion of BEV features ; Using real frames Generate a mask with the same size as the three BEV features mentioned above. ; mask Multiplying with each BEV feature by a dot product forms the foreground-aware BEV features, thereby calculating the teacher and student models. , , Feature loss, respectively , , The image BEV feature loss LiDAR BEV characteristics and multimodal fusion of BEV features The sum of the features constitutes the total BEV characteristic loss. ; Step 4: Construct a joint loss distillation strategy between the teacher model and the student model, and train them. A joint loss distillation strategy is constructed for the teacher model and the student model, resulting in the BEV feature loss from the knowledge distillation of the teacher model and the student model. and prediction result loss Total loss Thus, different knowledge is transmitted to the student model through multiple types of joint loss, and then the student model is trained under supervision. Step 5: Perform 3D target detection based on the student model after knowledge distillation. Based on pruning and distillation strategies, and a trained student model, a 3D object detection task is performed, where the data input, image processing, and enhancement methods are the same as in step one.

2. The method for accelerating 3D object detection using pruning and distillation according to claim 1, characterized in that: In step one, data processing includes image enhancement processing and point cloud data enhancement processing, both of which include cropping and rotation processing.

3. The method for accelerating 3D object detection using pruning and distillation according to claim 1, characterized in that: In step one, the image feature extraction network includes an image feature encoding module and an LSS-like view space transformation module.

4. The method for accelerating 3D object detection using pruning and distillation according to claim 1, characterized in that: In step one, the lidar point cloud feature extraction network includes a point cloud voxel feature encoder and a BEV feature extraction module.

5. The method for accelerating 3D object detection using pruning and distillation according to claim 1, characterized in that: In step two, the specific encoding method is as follows: S1: For specific modules The convolutional channels are used to encode binary genotypes to generate a population. One individual in the population is, , ; S2: Define the number of genetic iterations as... Next, the initial population... Through fitness function Calculate fitness value Based on the fitness value, the roulette wheel selection probability is calculated, and the roulette wheel selection probability method is used to select K individuals for the population. S3: Perform crossover and mutation operations on the K individuals with the best fitness values ​​using a genetic algorithm, and generate new individuals through crossover inheritance. A new individual is produced through genetic variation. A new individual, using inherited traits. A new population; S4: Denote the population as The population is then subjected to the next genetic iteration until an individual in the population has reached the desired fitness value or the maximum number of iterations has been reached.

6. The method for accelerating 3D object detection using pruning and distillation according to claim 5, characterized in that: The fitness function Defined as: In the formula, ACT is an activation function whose output range is from 0 to 1. , and These are heatmaps showing the categories before and after pruning. This represents the number of categories for the object detection task.

7. The method for accelerating 3D object detection using pruning and distillation according to claim 1, characterized in that: In step three, the specific method for constructing the prediction result loss is as follows: Extract the prediction results from the teacher model and the student model. Student model prediction Teacher model prediction Calculate loss ; predict students' models Compared with the true result GT loss Predictive loss from the teacher model And the loss relative to the actual result Add them together to form the total predicted loss .

Images