Lightweight target detection multi-stage hierarchical distillation method and equipment applied to automatic driving and storage medium

Abstract

The invention provides a lightweight target detection multi-stage hierarchical distillation method applied to automatic driving. The lightweight target detection multi-stage hierarchical distillation method comprises the following steps: acquiring a target detection data set; training teacher and student target detection models based on different parameter quantities by using the obtained data; performing multi-stage combined distillation according to the classification score of the teacher model, the position of the positioning frame and the multi-scale feature map; the distilled student model is applied to the sensing layer of the autonomous vehicle, and the target detection precision is improved under the condition that the parameter quantity is not additionally increased. The multi-stage hierarchical distillation method is designed, the detection precision of the lightweight target detection model can be effectively improved, efficient knowledge transmission from the teacher model to the student model is achieved, the balance problem of the lightweight model between precision and efficiency is effectively solved, the method is used for automatic driving, and the performance of an automatic driving system is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a lightweight multi-stage hierarchical distillation method, apparatus, and storage medium for target detection in autonomous driving. Background Technology

[0002] Autonomous vehicles rely on various sensors, such as LiDAR, cameras, and millimeter-wave radar, to perceive their surroundings. Among these, the visual information acquired by cameras is rich in detail and crucial for target recognition. However, target detection algorithms face a trade-off between model complexity and computational resources in real-time applications: while deep models offer high detection accuracy, their large number of parameters and computational cost make them difficult to deploy on edge devices; and while lightweight models (such as YOLOv8n) meet efficiency requirements, their accuracy drops significantly, especially for small targets and complex scenes. Knowledge distillation, by transferring knowledge from teacher models to guide student model training, has become a key technology for resolving this contradiction.

[0003] However, existing distillation methods still have significant limitations: In feature distillation, traditional methods typically align the intermediate layer features of the teacher and student networks directly, ignoring the semantic differences between features at different levels, leading to insufficient feature transfer for difficult samples such as small targets; in logit distillation, traditional methods, especially classification distillation which uses the Softmax scheme, conflict with the Sigmoid scheme actually used by detection models, and localization distillation relies on traditional IoU, which results in gradient vanishing when the predicted bounding boxes do not overlap, failing to effectively convey the relative position information of the bounding boxes. These limitations make it difficult for existing methods to significantly improve the detection accuracy of the student model while maintaining lightweight design, especially its adaptability to small targets and complex scenes. Based on this, a lightweight multi-stage hierarchical distillation method for object detection is proposed, which can be applied to autonomous driving. Summary of the Invention

[0004] This invention addresses the limitations of existing distillation methods by proposing an innovative multi-stage hierarchical distillation framework. Feature distillation is constructed using multi-scale feature maps and attention maps from different stages of the teacher model, and adaptive weights are used to dynamically adjust the loss weights at different stages. Simultaneously, a logit distillation strategy is employed, decomposing classification logits into multiple binary classification tasks and applying binary cross-entropy loss. For localization distillation, Wise-IoU loss and a dynamic focusing mechanism are introduced to address the gradient vanishing problem of traditional IoU, thereby optimizing the accuracy of bounding box regression. This scheme, through the synergistic optimization of feature distillation and logit distillation, significantly improves the detection accuracy of the lightweight student model (YOLOv8n) while maintaining high inference speed, particularly enhancing its adaptability to small targets and complex scenes. It effectively solves the key problem of insufficient accuracy in lightweight models for object detection, and its application in autonomous driving improves the performance of autonomous driving systems.

[0005] In a first aspect, the present invention provides a lightweight multi-stage hierarchical distillation method for target detection applied to autonomous driving, comprising the following steps: Obtain object detection datasets suitable for autonomous driving scenarios; Train teacher and student target detection models with different parameter values ​​using the acquired data; Multi-stage joint distillation is performed using the classification scores, bounding box positions, and multi-scale feature maps of the teacher model. The distilled student model is applied to the camera perception layer of autonomous vehicles to improve target detection accuracy without increasing the number of parameters.

[0006] In one optional implementation, the target detection dataset suitable for autonomous driving scenarios is the VOC dataset, and the model is trained using the VOC2007+VOC2012 training set, and the performance is evaluated using the VOC2007 test set.

[0007] In one optional implementation, the teacher model is YOLOv8s and the student model is YOLOv8n. The YOLOv8s and YOLOv8n model structures include a backbone network, a neck network, and a detection head. The backbone network adopts a C2f module, the neck network contains a PAN-FAN structure, and the detection head adopts an anchor-free detection method with a decoupled head structure.

[0008] In one optional implementation, the multi-stage joint distillation includes feature distillation, logit classification distillation, and Wise-IoU localization distillation. Feature distillation uses multi-layer feature maps and attention maps of the teacher network to guide students. Logit classification distillation transforms the classification logit mapping into multiple binary classification mappings. Localization distillation replaces the traditional IoU with Wise-IoU.

[0009] In one alternative implementation, the feature distillation dynamically allocates feature distillation weights for the backbone, neck, and head using Softmax, and uses the focus attention module to generate a spatial attention map of the teacher model, guiding the student model to focus on key target areas such as those in autonomous driving scenarios.

[0010] In one optional implementation, the attention module generates an attention map through average pooling, max pooling, and 1×1 convolution. The loss function of the attention module is: , in, and These represent the feature maps of the teacher model and the student model in the backbone network, respectively. , and , These represent feature maps of the teacher model and the student model in the neck network and the detection head network, respectively. parameter , , This indicates the weights of the distillation of features in the skeletal, neck, and head regions; , A spatial attention weight map representing the teacher and student models; The hyperparameter is typically set to 1.

[0011] In one alternative implementation, the classification logits are mapped to multiple binary classification maps, and the distillation loss is calculated based on these binary classification maps. The specific formula for calculating the classification distillation loss is as follows: , in, The total number of samples, =1-20, target category index; , For student and teacher models for the first The sample, the first Binary classification probability of the class.

[0012] In one alternative implementation, the location distillation loss function is: , in, The total number of samples, The Wise-IoU value is the bounding box value of the teacher and student models.

[0013] In a second aspect, the present invention also provides an electronic device, comprising: a processor and a memory; the memory storing computer execution instructions; the processor executing the computer execution instructions stored in the memory, causing the electronic device to perform the method described above.

[0014] Thirdly, the present invention also provides a computer stored program, which stores a computer program that, when executed by a processor, implements the method described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention designs a multi-stage hierarchical distillation method, which can effectively improve the detection accuracy of lightweight target detection models, realize efficient knowledge transfer from teacher models to student models, effectively solve the problem of balancing accuracy and efficiency in lightweight models, and significantly improve the performance of autonomous driving systems when applied to autonomous driving.

[0016] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0017] Figure 1 A flowchart of the distillation method provided by the present invention; Figure 2 This is a schematic diagram of the YOLOv8 enhanced hierarchical weighted feature distillation of the present invention; Figure 3 This is a schematic diagram of the attention-focusing module of the present invention; Figure 4 This is a structural diagram of the classification and positioning frame distillation method based on Logit in this invention. Detailed Implementation

[0018] To facilitate understanding of the present invention, a more comprehensive description of the present invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the present invention. However, the present invention can be implemented in different forms and is not limited to the embodiments described in the text. Rather, these embodiments are provided to make the disclosure of the present invention more thorough and complete.

[0019] Reference Figure 1-4 This invention proposes a multi-stage hierarchical distillation method based on lightweight target detection, comprising the following steps: S1. Obtain the object detection dataset. Specifically, the dataset includes, but is not limited to, the VOC dataset, a commonly used object detection dataset first released by a computer vision team at Oxford University in 2005. It is primarily used for small object detection and is widely applied in academic research and industrial practice. This dataset contains 20 different object categories, including (airplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorcycle, person, potted plant, sheep, sofa, train, and television monitor). The data annotation is detailed and of high quality. This dataset is suitable for detecting pedestrians, vehicles, and other objects in autonomous driving scenarios. Different datasets can be selected as needed. We use the VOC2007+VOC2012 training set for model training and evaluate the model performance on the VOC2007 test set to ensure the fairness and reproducibility of the results. The entire dataset contains 16,551 training images and 4,962 test images, and provides detailed bounding box annotation information to support accurate evaluation of the object detection task.

[0020] S2. Train teacher and student object detection models with different parameter sets using the acquired data. Specifically, YOLOv8 is an object detection algorithm launched by Ultralytics, which continues the design philosophy of the YOLO series and has been optimized in several aspects. Its model structure mainly consists of three parts: backbone, neck, and head. The backbone uses the C2f module to replace the C3 module in YOLOv5, and enhances the model performance by optimizing gradient flow. The neck introduces a PAN-FAN structure, including two PAN modules and one SPPF module, for multi-scale feature fusion, which improves the model's multi-scale detection capability. The head adopts a decoupled head structure, separating the regression branch and the classification branch, and converting from anchor-based to anchor-free detection method, which improves the accuracy and flexibility of detection, thereby adapting to the real-time requirements of autonomous driving.

[0021] S3. Multi-stage joint distillation is performed using the classification scores, bounding box positions, and multi-scale feature maps of the teacher model. The multi-stage hierarchical distillation framework introduced in this invention includes three parts: enhanced hierarchical weighted feature distillation, binary classification distillation, and Wieou bounding box distillation. During the training phase, a strategy of co-optimization of feature distillation and logits distillation is adopted: guiding signals are generated through the multi-layer feature maps and attention maps of the teacher model (yolov8s), and the distillation loss weights of different levels of features are dynamically adjusted using adaptive weight allocation; at the same time, the classification logits of the teacher model are decomposed into multiple binary classification tasks for class-by-class distillation, and the Wise-IoU loss combined with a dynamic focusing mechanism is used to optimize the localization distillation. The teacher model provides stable supervision by freezing parameters, and the student model (yolov8n) receives joint supervision from the original ground truth labels and the teacher's soft labels. During the inference phase, only the lightweight student model optimized by multi-stage distillation is retained, which significantly improves detection accuracy while maintaining efficient inference speed.

[0022] In the proposed enhanced hierarchical weighted feature distillation method, feature distillation involves extracting feature maps from the backbone, neck, and head of the network and applying soft weights to these feature maps to automatically adapt to the representational power of each part. Specifically, a softmax operation is used to assign weights to different feature layers, thereby dynamically adjusting the importance of each layer. Then, the Kullback-Leibler (KL) divergence is used to measure the difference between the teacher and student models at the feature level, thereby calculating the distillation loss. KL is commonly used to measure the distributional difference between data, and our goal is to minimize the KL difference between the teacher and student models. The overall enhanced hierarchical weighted feature structure diagram is shown below. Figure 2 As shown. The trunk, neck, and head provide feature map information for characteristic distillation.

[0023] To further improve the distillation effect, a focused attention mechanism is added before feature distillation to calculate the attention map of the teacher feature map. The focused attention module extracts the spatial information of the input features through average pooling and max pooling. The entire dataset contains 16,551 training images and 4,962 test images. The two are concatenated and then input into a 1×1 convolution for feature transformation. Finally, a normalized spatial attention map is generated through Sigmoid activation. Specifically, the feature map of the teacher model is first processed... Calculate a spatial attention weight graph This highlights regions with strong representational power in the image. The teacher's attention map represents the degree of attention the model focuses on at different spatial locations. Then, the teacher's attention map is used as guiding information to perform weighted distillation on the student model's feature map, enabling the student to more accurately learn the teacher's model's representational power in important regions. Furthermore, we use KL divergence loss to measure the difference between the teacher's and student's attention maps, prompting them to focus on key regions consistent with the teacher's (such as the road ahead or intersection targets in autonomous driving). The attention module is specifically designed as follows: Figure 3 As shown. The attention-guided feature distillation loss can be represented as follows: (1), (2), in, and These represent the feature maps of the teacher model and the student model in the backbone network, respectively. Similarly, , and , These represent feature maps for the neck network and the detection head network, respectively. Parameters , , This represents the weights of the feature distillation loss at the three network locations mentioned above. The hyperparameter is typically set to 1.

[0024] (3), , , These represent the number of branches in the backbone, neck, and head networks, respectively. This represents the Softmax operation. This method dynamically changes the contribution of features at different levels to distillation losses, further improving distillation efficiency.

[0025] In binary classification distillation, due to the significantly larger number of background pixels compared to foreground pixels in small and dense object detection tasks, severe foreground-background sample imbalance easily arises during training. To alleviate this problem, YOLOv8 employs the Sigmoid scheme for classification, allowing the score of each class to be optimized independently, rather than establishing dependencies between classes. Sigmoid more effectively handles foreground-background sample imbalance by transforming a multi-class classification task into multiple binary classification tasks. Therefore, if YOLOv8 directly adopts the traditional Softmax classification distillation technique, it will lead to inconsistencies in the scheme across tasks, affecting distillation performance. To address this issue, we introduce a binary classification distillation strategy, mapping classification logits to multiple binary classification mappings, and calculating the distillation loss based on these binary classification mappings, such as... Figure 4 As shown, the specific formula for calculating classification loss is as follows: (4), (5), in, The total number of samples, =1-20, target category index; , For student and teacher models for the first The sample, the first The binary classification probability of a class. Therefore, class distillation can improve the accuracy of target class recognition in autonomous driving scenarios.

[0026] The WIoU localization distillation method transfers localization knowledge from the teacher model to the student model, further improving localization accuracy. IoU (Intersection over Union) is used to evaluate the overlap between two bounding boxes, which we use as the objective of distillation. We obtain localization predictions from both models and decode them into bounding boxes based on anchor point positions. Then, we calculate the IoU between the predicted bounding boxes and use it as part of the distillation loss. The goal is to maximize the IoU between the teacher and student models to improve the consistency of bounding box predictions. However, traditional IoU has a drawback: when the two predicted boxes do not overlap... The gradient vanishing phenomenon fails to reflect the actual distance and potential information between boxes. To address this issue, and to focus more on samples that are difficult to predict rather than giving excessive weight to samples that are predicted well, we use Wise-IoU as the loss function for IoU distillation. The core principle of Wise-IoU is that, in addition to calculating the standard IoU value, it introduces a penalty term based on the center point distance and the size of the bounding box. When the center point distance between the predicted box and the target box is large, Wise-IoU amplifies the loss value, thereby encouraging the model to align the center point more accurately. This adjustment is achieved through an exponential function that converts the center point distance into a scaling factor and combines it with the basic IoU loss. Equation (6) shows the construction of Wise-IoU, where Used to enhance attention to difficult target boxes with poor overlap. As a monotonic focusing factor, it effectively reduces the contribution of easy-to-learn samples to the loss, making the model more focused on difficult samples. Represents the IoU loss separated from the computation graph. This represents a moving average with momentum, ensuring that the monotonic focus factor remains at a high level, thereby preventing the convergence speed from being too slow in the later stages of training. It is a hyperparameter that is generally greater than 1.

[0027] (6), (7), in, , This represents the coordinates of the center point of the student model's bounding box; , This represents the coordinates of the center point of the teacher model's bounding box; W and H represent the width and height of the outer bounding box; Therefore, the formula for calculating the distillation loss of the positioning frame is as follows: (8), (9), in Represents the anchor point. Predicted offsets for teachers and students This represents the bounding box coordinates after decoding. Through Calculated.

[0028] Therefore, Wise-IoU positioning distillation can improve the accuracy of target localization in autonomous driving scenarios.

[0029] In summary, the novel distillation methods proposed in this paper include feature map distillation, binary classification distillation, and Wise-IoU-based localization distillation. Feature map distillation aims to allow the student model to learn knowledge from feature layers of different scales in the teacher model; binary classification distillation is optimized for classification tasks; and IoU distillation is specifically designed for localization tasks. The overall distillation loss function is shown below: (10) in, , and These are hyperparameters used to control the weights of characteristic distillation, classification distillation, and localization distillation, respectively.

[0030] Finally, in S4, the distilled student model is applied to the perception layer of the autonomous vehicle to improve object detection accuracy without increasing the number of parameters. Specifically, in our experiments, we use YOLOv8N as the student model and YOLOv8S as the teacher model. The training method is consistent with YOLOv8, including using stochastic gradient descent (SGD) with momentum, step-wise decay of the learning rate, and a warm-up strategy. To improve training performance, we adopted two powerful data augmentation techniques—Mosaic and Mixup—both of which are based on previous research. The learning rate was set to 0.01, momentum to 0.937, and weight decay coefficient to 0.0005. A complete list of hyperparameters can be found in the YOLOv8 code. All experimental results were obtained by training on two NVIDIA 3080Ti GPUs. During inference, we used the same settings as YOLOv8, with the IoU threshold set to 0.7 and the image input size to 640. Other parameters are not listed separately.

[0031] The present invention will now be described in detail with reference to specific embodiments. The present invention uses NVIDIA GPU graphics cards as the computing platform and PyTorch as the network model framework.

[0032] 1. Dataset Acquisition and Introduction Log in to the website: Download the VOC dataset to your local machine from http: / / host.robots.ox.ac.uk / pascal / VOC / voc2012 / .

[0033] In the experiments, we trained the model using the VOC2007+VOC2012 training sets and evaluated its performance on the VOC2007 test set to ensure the fairness and reproducibility of the results. The entire dataset contains 16,551 training images and 4,962 test images, and provides detailed bounding box annotations to support accurate evaluation of the object detection task.

[0034] 2. Model Training We use YOLOv8n as the student model and YOLOv8s as the teacher model. During distillation, the teacher model provides stable supervision by freezing parameters, while the student model (yolov8n) receives joint supervision from both the original ground truth labels and the teacher's soft labels.

[0035] 3. Model Evaluation Based on the prediction results after the above processing, mAP@0.5:95 was used to measure the model's performance. Table 1 shows the comparative experimental results of this invention with other models on the VOC dataset. Table 2 compares this distillation method with other distillation methods.

[0036] Table 1. Performance comparison of different models on the VOC dataset.

[0037] Table 2 Comparison with different knowledge distillation methods

[0038] In summary, this invention designs a multi-stage hierarchical distillation method with fewer parameters and higher computational efficiency, which can effectively improve the detection accuracy of lightweight target detection models. It realizes efficient knowledge transfer from teacher models to student models and effectively solves the problem of balancing accuracy and efficiency in lightweight models. When applied to autonomous driving, vehicles need to identify and locate various target objects in the surrounding environment in real time and accurately, such as pedestrians, other vehicles, traffic signs and traffic lights. This places extremely high demands on the accuracy and efficiency of target detection. The distillation method proposed in this invention can meet these requirements and significantly improve the performance of autonomous driving systems.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight, multi-stage hierarchical distillation method for target detection applied to autonomous driving, characterized in that, Includes the following steps: Obtain the object detection dataset; Train teacher and student target detection models with different parameter values ​​using the acquired data; Multi-stage joint distillation is performed using the classification scores, bounding box positions, and multi-scale feature maps of the teacher model. The distilled student model is applied to the perception layer of autonomous vehicles to improve target detection accuracy without increasing the number of parameters.

2. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 1, characterized in that: The target detection dataset is the VOC dataset. The model is trained using the VOC2007+VOC2012 training set, and the performance is evaluated using the VOC2007 test set.

3. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 1, characterized in that: The teacher model is YOLOv8s, and the student model is YOLOv8n. The YOLOv8s and YOLOv8n model structures include a backbone network, a neck network, and a detection head. The backbone network uses C2f modules, the neck network contains a PAN-FAN structure, and the detection head adopts an anchor-free detection method with a decoupled head structure.

4. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 1, characterized in that: The multi-stage joint distillation includes feature distillation, logit classification distillation, and Wise-IoU localization distillation. Feature distillation uses multi-layer feature maps and attention maps of the teacher network to guide students. Logit classification distillation transforms the classification logit mapping into multiple binary classification mappings. Localization distillation replaces the traditional IoU with Wise-IoU.

5. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 4, characterized in that: The feature distillation dynamically allocates feature distillation weights for the backbone, neck, and head using Softmax, and uses the focus attention module to generate a spatial attention map of the teacher model to guide the student model to focus on key target regions.

6. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 5, characterized in that: The attention module generates an attention map using average pooling, max pooling, and 1×1 convolution. The loss function for the attention module is: ， in, and These represent the feature maps of the teacher model and the student model in the backbone network, respectively. , and , These represent feature maps of the teacher model and the student model in the neck network and the detection head network, respectively. parameter , , This indicates the weights of the distillation of features in the skeletal, neck, and head regions; , A spatial attention weight map representing the teacher and student models; The hyperparameter is typically set to 1.

7. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 4, characterized in that: The classification logits are mapped to multiple binary classification mappings, and the distillation loss is calculated based on these binary classification mappings. The specific formula for calculating the classification distillation loss is as follows: ， in, The total number of samples, =1-20, target category index; , For student and teacher models for the first The sample, the first The binary classification probability of the class.

8. The lightweight target detection multi-stage hierarchical distillation method for autonomous driving according to claim 4, characterized in that: The location distillation loss function is: ， in, The total number of samples, The Wise-IoU value is the bounding box value of the teacher and student models.

9. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1 to 8.

10. A computer stored program, characterized in that, The device contains a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images