Lightweight target detection knowledge distillation method
Through the hand-guided distillation framework, multi-level features are extracted and feature matching and loss optimization is performed for the target detection task, which solves the problem of degradation of distillation accuracy of target detection knowledge in the existing technology, and achieves efficient goals in a low computing resource environment Test.
Patent Information
- Application Number
- CN202410169440.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-05
- Publication Date
- 2024-12-06
AI Technical Summary
The existing object detection knowledge distillation technology cannot effectively reduce the accuracy loss after model compression, especially in detection tasks in multi-scale features and interference contexts, resulting in poor practical application deployment results.
Using a distillation framework based on step-by-step guidance, multi-level features are gradually extracted through feature matching between teacher networks and student networks, multi-stage feature imitation loss measurement and reweighted step-by-step distillation loss, helping students' models understand the nature of the task and accelerate convergence, and reducing accuracy losses. .
It significantly improves the compression accuracy of the object detection model, reduces resource overhead, improves the deployment performance of the model in a low computing resource environment, and enhances the real-time and accuracy of the detection tasks.
Smart Images

Figure CN119089950A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision and pattern recognition, and specifically relates to a lightweight target detection knowledge distillation method that can be deployed in a low computing resource environment, including edge devices, ordinary computer terminals, etc. Background Art
[0002] In recent years, with the rapid development of social, live broadcast, and short video platforms, multimedia data has exploded. Thanks to this, the development of multimedia technology has also ushered in a golden age. In the field of image processing and computer vision, the continuous improvement of computer hardware performance has made convolutional neural networks (CNN) a research hotspot. Among them, target detection, as one of the basic problems of computer vision, aims to find all objects of interest in the image, determine their location range and category, and subsequently use this high-level semantic information to understand the scene. Object detection is also the basis of many advanced application tasks, such as pedestrian re-identification, face detection, vehicle detection, behavior analysis, autonomous driving, and so on. For example, when the security system is performing pedestrian re-identification tasks, it needs to first use pedestrian detection technology to extract pedestrians separately; in order to judge the road conditions in autonomous driving, the visual subsystem needs to first detect the vehicles or pedestrians in front of it.
[0003] For identity verification at the railway entrance, the passenger's face needs to be cut out from the captured image first.
[0004] Unlike coarse-grained image classification tasks, object detection tasks require analyzing the fine-grained categories and locations of each object in an image. Under the influence of changing shooting angles, object postures, lighting, and mutual occlusion of objects, detection tasks often encounter huge challenges. The earliest researchers used traditional methods such as extracting manual features to solve this problem, which usually includes three specific methods: region selection, feature extraction, and classification regression.
[0005] However, the subsequent traditional models have entered a bottleneck period except for a slight improvement by relying on multi-system integration. The main reason is that the semantics of manual features are poor and the preset sliding window position is not accurate, resulting in low system efficiency and poor accuracy. Since CNN's first success in classification tasks, various CNN-based detectors have appeared in the field of object detection, and the effect is greatly improved over previous traditional methods. Various difficulties have been continuously overcome, and the accuracy of object detectors has been continuously improved. At present, the most advanced model can achieve more than 80% mAP on the PASCAL VOC dataset, and more than 60% AP50 and more than 45% AP (0.5:0.95) on the MS COCO dataset.
[0006] Although models based on convolutional neural networks have achieved great results in the field of target detection compared to traditional methods, researchers often pursue extreme results and design deeper and more complex networks to continuously refresh various records. When it comes to actual applications, developers often suffer from the huge memory and computing resource overhead brought by deep learning models. Because the prediction of a picture involves the access of tens of millions or even hundreds of millions of parameters, and requires tens of billions of floating-point multiplication operations. Even though the performance of GPU servers is already very powerful, applications can rely on the server for inference calculations, but facing the daily processing volume of tens of millions of video images on the Internet is still stretched, and for developers, service costs are also an issue that must be paid attention to. What is more worrying is that most of the current intelligent application devices often do not have the powerful computing power of GPUs, such as embedded devices, mobile devices, general-purpose PCs, industrial equipment, etc., which are often the main devices used in industries such as mobile intelligent services, self-driving cars, industrial detection, intelligent security, drones, and the Internet of Things. For example, first-order detectors such as YOLO and SSD can process nearly 100 image inputs per second on a single GPU, but the computing burden is too heavy for low-end devices, resulting in very slow processing. In addition, the target detection task itself has high real-time requirements, which will cause the hardware cost of related industries to rise, and then lead to the model's inability to fully exert its performance. These problems have brought layers of obstacles to the actual application and deployment of the detection model.
[0007] In response to the above problems, researchers began to use model compression technology to design lightweight models to reduce model calculations and memory resource consumption. Unlike other model compression methods that often require demanding hardware or software environments to achieve inference acceleration, knowledge distillation is a simpler and more direct method that aims to grant the rich prior knowledge contained in the large model to the small model with poor capabilities, thereby assisting the training process. It is much better than directly training the small model, and ultimately the powerful student network is directly used for actual deployment without additional restrictions. However, current target detection knowledge distillation technology is often extended from classification distillation technology and cannot grasp the essence of the target detection task.
[0008] As we all know, target detection is a more difficult task than classification. It not only requires analyzing object category information, but also relies on multi-scale and accurate fine-grained features to analyze object location information. In addition, the interference caused by irrelevant background objects is also greater. Therefore, most model compression methods cannot really work on detection tasks, and the accuracy is also greatly reduced, which greatly reduces the experience of actual application deployment. Considering that the knowledge distillation model for target detection is not fully mature, there is still room for in-depth exploration and improvement. Summary of the invention
[0009] In view of the problems that the existing knowledge distillation method ignores the feature hierarchy in the detector network and the sensitivity of the detector to multi-scale features, resulting in serious loss of accuracy after compression, the present invention provides a distillation framework based on hand-in-hand guidance, trying to improve the knowledge distillation method for target detection. It aims to apply more comprehensive hand-in-hand guidance to the student network, further taking into account the excellent performance of the detector and less resource overhead. The main purpose is to reduce the accuracy loss of the target detection model after compression. In general, the knowledge is gradually extracted in the entire feature extraction stage of the teacher network for the knowledge distillation of the small model, helping it to understand the essence of the task and accelerate convergence. The core of the invention is to perform comprehensive distillation of multi-level features. Specifically, first, under the condition of semantic compatibility, feature sampling and feature matching are performed on the teacher network and the student network to establish a one-to-one correspondence; then the distance loss between the corresponding features is reasonably calculated, and the student model is allowed to imitate the features; for excessive feature processing, in order to ensure that the distillation learning of a specific part can be strengthened in a targeted manner, a weight generation that highlights the focus and a re-weighted loss optimization algorithm are proposed to focus the overall distillation target on the focus; finally, the comprehensive classification, positioning, and distillation multiple losses are jointly optimized.
[0010] In order to achieve the above objectives, the present invention divides the method into five stages: the stage of establishing a hand-in-hand distillation framework; the stage of matching the features of the teacher network and the student network; the stage of measuring the multi-stage feature imitation loss; the stage of re-weighted hand-in-hand distillation loss; and the stage of distillation training and model deployment prediction. The implementation steps of each stage are:
[0011] 1. Step-by-step distillation framework establishment stage:
[0012] The present invention improves the current knowledge distillation method which mainly focuses on the problem of high-level features or soft label imitation, and adds more shallow feature semantic knowledge extraction as the basis for high-dimensional feature learning. During the distillation training process, the teacher network guides the student network step by step to optimize the training, so that the student network can learn more prior knowledge and better achieve the training goals. The core of the distillation framework proposed in the present invention is to perform comprehensive distillation of multi-level features. In a sense, it is a more comprehensive regularization method with the help of the teacher network.
[0013] 2. Teacher network and student network feature matching stage:
[0014] In terms of distillation object setting, the present invention selects the standard one-stage detector SSD based on VGG16 as the teacher network, and uses the VGG16 SSD model with half the number of channels in each layer. It has a similar structure to the teacher network, but only has half the number of channels, called channel-half SSD. This student model has faster prediction speed, less computational consumption, and is more suitable for deployment. Of course, other small models are also applicable, such as MobileNet, VGG11, etc., mainly due to the more compatible feature matching strategy.
[0015] After preparing the teacher network and the student network, the present invention combines the characteristics of the two networks Figure 1 Once matched, they are used for subsequent distillation. Considering that the use of all features will inevitably lead to excessive memory consumption and slow convergence, and the number and size of feature maps in the teacher network and the student network may be different. Therefore, in the feature extraction stage, the present invention samples each stage dimension of all feature maps to ensure semantic correspondence. Then use the transformation layer to adjust the size to ensure the consistency of the feature map size. The transformation layer includes 1x1 convolution, bilinear upsampling and downsampling. Overall process
[0016] The brief description is as follows:
[0017] x′ imit =adaption(select(x stage ))
[0018] where x stage Represents the original feature map of each stage, x′ imit Represents the generated feature map to be matched,
[0019] select and adaptation represent the selection and adaptation transformation processes respectively. Note that if the number of stages does not match, the early unmatched stages are selectively ignored. Finally, the teacher network and student network feature maps with the same number and size are obtained, which can be matched one-to-one.
[0020] 3. Multi-stage feature imitation loss measurement stage:
[0021] After establishing the feature matching between the teacher and student networks, it is necessary to calculate the feature distance loss to ensure that the student model is optimized towards the teacher model under distillation supervision. In order to solve the problem that the L2 distance optimization between the student feature vector v and the teacher vector z in the multi-stage hand-in-hand distillation framework is too harsh, this paper improves the feature imitation loss and compares the student feature vector v at each position with the teacher vector z.
[0022] The cosine distance between z is used as the imitation loss function:
[0023]
[0024] Because it is difficult to force the student network to be exactly the same as the teacher network at every stage. Therefore, the present invention adopts a slightly weaker cosine distance metric L imit_cosine , which can avoid excessive pursuit of consistency. The improved cosine similarity loss function aims to capture the vector direction of the feature, while strengthening the guidance through more multi-stage imitation, which is still a strong constraint overall.
[0025] Combining the feature vectors of each stage and each position, the feature imitation loss proposed by the present invention is:
[0026]
[0027] Where S is the total number of stages of feature extraction, Hs and Ws are the height and width of the feature map at each stage, j and j represents the student feature vector and teacher feature vector at the (i, j) position in the sth stage. The Limit imitation loss uses the cosine metric of Limit_cosine.
[0028] 4. Reweighted Hands-On Distillation Loss Stage:
[0029] Even if the network feature graph is sampled in feature matching, there are still too many feature vectors to be imitated at each stage and position, making it difficult to grasp the key points in distillation training. Therefore, the present invention introduces a reweighted hand-in-hand distillation imitation loss from both micro and macro perspectives to make distillation more targeted, as shown below:
[0030]
[0031] The added us in the formula represents the weight of each stage from a macro perspective. j represents the microscopic spatial weight of the (i,j) position at the sth stage. In addition, the total feature loss function of each stage will be ∑i,j j is normalized, because different image inputs may result in completely different spatial weight distributions. By comparing the original loss function, it can be seen that after the vector imitation loss at each position is calculated, it will be multiplied by a micro weight factor, and then the losses at each position will be normalized as the sum of the losses at each stage when accumulated, and then multiplied by a macro weight factor for weighted sum to obtain an improved reweighted imitation loss function. A natural question is how to obtain these weights. The present invention proposes to evaluate the importance from both the macroscopic overall and microscopic local aspects, and proposes some weight generation strategies: learnable attention module automatic evaluation; use focal weights to make the loss distribution more balanced; use feature mean or variance statistics to evaluate importance, the larger the mean / variance value, the larger the corresponding weight; use GT-mask to focus on key foreground areas, which can reduce the influence of irrelevant noise from the background. The present invention experimentally tried to combine these strategies and found that the best effect can be achieved by using GT-mask to focus on foreground areas at the micro level and focusing on stages with large variance statistics at the macro level.
[0032] 5. Distillation training and model deployment prediction stage:
[0033] In distillation training, the above-mentioned imitation distillation loss is combined with the cross entropy classification loss Lc and Smooth L1 positioning loss Ll commonly used in detection to construct the multi-task loss as follows:
[0034] L=L c +λ1L l +λ2L i
[0035] Optimize the parameters of the student network together, where λ1 and λ2 are the weight coefficients of the loss terms. Adjust this coefficient to keep all loss terms at the same order of magnitude. In order to speed up convergence during training, the student model will be allowed to self-study for several rounds to have some detection foundation before distillation training. In addition, the number of iterations is slightly increased in distillation training to ensure that there are enough training rounds after the introduction of distillation supervision. When training is completed and ready for deployment, all additional parts except the original student model will be removed, including the adaptive transformation module added to the student network to facilitate feature matching. When performing target detection after deployment, the given image to be detected is input into the trained student model according to a certain preprocessing method. The model will eventually output multiple fixed-size outputs, each of which includes a category label and a position vector, representing the target position and target category contained in the image.
[0036] In general, by summarizing the technical features of the present invention, compared with the existing most advanced technology, it has the following advantages and beneficial effects:
[0037] (1) The main innovation of this method is that it proposes a new distillation architecture for the challenging object detection task, namely, a distillation method based on step-by-step guidance, which aims to extract detailed and refined prior knowledge, help small models understand the essence of the task and accelerate convergence, thereby improving the effect of distillation.
[0038] (2) Aiming at the proposed hands-on distillation method, a series of detailed feature imitation strategies are proposed to achieve the best results, including a more reasonable feature matching strategy, an improved imitation loss function, a weight generation that highlights the key points, and a re-weighted optimization algorithm. These strategies complement each other and enable the overall distillation framework design to achieve better results.
[0039] (3) This framework relies only on the feature maps in the network instead of the anchor boxes, labels, candidate set extraction networks and other modules in the detector, which makes it more versatile. In addition, direct end-to-end training is relatively simple and the model deployment is completely consistent with the original model without any additional restrictions. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 Optimize the concept map of the hand-in-hand distillation model in the present invention;
[0041] Figure 2 This is a comparison chart of the improved distillation method of the present invention;
[0042] Figure 3 It is a schematic diagram of feature matching in the present invention;
[0043] Figure 4 This is a visualization diagram before and after the hand-to-hand distillation in the present invention; DETAILED DESCRIPTION
[0044] In order to more clearly and meticulously introduce the technical details, method advantages and innovations of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is hereby noted that the specific application embodiments introduced herein are only used to explain the present invention and are not limited to the present invention. In addition, the technical features of the various embodiments of the present invention described below may be combined with each other when there is no conflict between the embodiments.
[0045] Compared with the problem of insufficient constraints in current target detection distillation methods, the present invention designs a more comprehensive Hands-on Guidance Distillation (HGD) framework. First, distillation preparations for feature matching at each level are performed, and then the cosine metric between matching feature pairs is calculated. Next, a moderate feature imitation loss function is constructed, and a targeted reweighting strategy is designed to make the distillation optimization more purposeful. Finally, multiple losses are combined to perform joint training of detectors.
[0046] The following will specifically explain the establishment of the hand-to-hand distillation framework, the matching of teacher network and student network features, the multi-stage feature imitation loss measurement, the re-weighted hand-to-hand distillation loss, and the distillation model training deployment pre-
[0047] The implementation steps of the five stages are as follows:
[0048] 1. Step-by-step distillation framework establishment stage:
[0049] Regarding the setting of the distillation framework, previous knowledge distillation methods mainly focused on high-level features or soft label imitation. The present invention analyzes and finds that early features are also meaningful for the extraction of fine-grained position information, and shallow features are the basis for high-dimensional feature calculation. That is, during the distillation training process, the teacher network gradually guides the student network to perform training optimization. Thanks to this, the student network can solve challenging problems step by step. Figure 1 Compared with the previous distillation methods, if we can provide guidance and supervision step by step, we can learn the working route and get more prior knowledge.
[0050] With the guidance of knowledge, it can eventually converge better and finally reach the supervision goal of the data set and solve more advanced problems.
[0051] Inspired by the continuous development of related knowledge distillation methods, this paper mainly performs extreme feature imitation on the full-stage feature graph of the network, such as Figure 2 As shown, it can be seen that the design difference between the present invention (d) and other methods (a), (b), (c) and so on. The present invention aims to gradually extract knowledge from the entire feature extraction stage of the network for knowledge distillation of small models. The core of this method is to perform comprehensive distillation of multi-level features, which is especially important for feature-sensitive target detection tasks, which is equivalent to imposing more supervision restrictions.
[0052] In a sense, it is a more comprehensive regularization method with the help of the teacher network.
[0053] 2. Teacher network and student network feature matching stage:
[0054] (2.1) Model preparation: The model compression process of knowledge distillation requires the participation of two models, one large and one small. For convenience, in terms of the distillation object, the present invention selects the standard one-stage detector SSD based on VGG16 as the teacher network. In contrast, the student network uses a small model with slightly lower accuracy but faster speed. Therefore, the present invention uses an SSD model with half the number of channels in each layer. It has a similar structure to the teacher network, but only half the number of channels, called channel-half SSD. Of course, other student models are also applicable, such as MobileNet, VGG11, etc., mainly due to the more compatible matching strategy.
[0055] (2.2) Feature matching between the teacher network and the student network: After preparing the teacher network and the student network in (2.1), the present invention first establishes a one-to-one correspondence between the student network and the teacher network to prepare for subsequent distillation. During training, considering that the use of all features will inevitably lead to excessive memory consumption, slow convergence, and the number of feature maps in the teacher network and the student network may not be equal. Therefore, this paper samples according to the stage dimension in the feature extraction stage to ensure semantic correspondence.
[0056] Specifically, this paper divides all feature maps into different stages (or blocks) according to the spatial size, and selects the last feature map of each stage, which contains the richest semantics of each stage, which is equivalent to using key features as representatives of each extraction stage. Then, the student network and the teacher network are paired with these sampled representatives of each stage. For the case of incompatible channel numbers, a full convolution transformation layer is added after the corresponding student features to perform adaptive transformation. Figure 3 , this section selects two stages for example. Before matching, each stage will sample a feature and perform adaptive transformation. The adaptive transformation operation includes 1x1 convolution, bilinear upsampling and downsampling.
[0057] The entire feature selection process can be summarized as follows:
[0058] x′ imit =adaption(select(x stage ))
[0059] where x stage Represents the original feature map of each stage, x′ imit Represents the generated feature map to be matched,
[0060] Select and adaptation represent the selection and adaptive transformation processes respectively. It should be noted that for the special case of input resolution mismatch, bilinear interpolation can be used for upsampling or downsampling adaptive transformation to ensure uniform resolution, or some shallow mismatched stages can be ignored as an alternative. After the above compatibility transformation, the features of the same size after processing by the teacher network and the student network can be Figure 1 One to one correspondence.
[0061] 3. Multi-stage feature imitation loss measurement stage:
[0062] After establishing the corresponding relationship of features and making compatible transformations, it is necessary to calculate the distance loss between features to ensure that the student model is optimized towards the teacher model under supervision. This is also the core step of knowledge distillation. Generally speaking, many methods use the L2 distance between the student feature vector v and the teacher vector Z at each position as the imitation loss function Limit_l2 :
[0063]
[0064] However, the present invention believes that such a distillation method is too strict, especially in the above-mentioned hand-in-hand distillation framework, because it is difficult to force the student network to be exactly the same as the teacher network at every stage.
[0065] Therefore, the present invention adopts a slightly weaker cosine distance metric L imit_cosine , avoid excessive pursuit of consistency:
[0066]
[0067] This cosine similarity loss function aims to grasp the vector direction of the feature, and at the same time strengthens the guidance through more multi-stage imitation, and is still a strong constraint overall. Combining the feature vectors of each stage and each position, the feature imitation loss proposed by the present invention is:
[0068]
[0069] Where S is the total number of stages of feature extraction, Hs and Ws are the height and width of the feature map at each stage, respectively. s ,j and Represent the student feature vector and teacher feature vector at the (i, j) position in the sth stage respectively. The Limit imitation loss uses the cosine metric of Limit_cosine. Through this imitation loss function, distillation optimization can be performed end-to-end during training.
[0070] 4. Reweighted Hands-On Distillation Loss Stage:
[0071] Even if the feature graph is sampled, the number of feature stages and positions in the network is still relatively large. It can be observed that the number of features to be imitated is still quite large, so it is difficult to grasp the key points in knowledge distillation. This paper introduces a re-weighted hand-in-hand distillation imitation loss from both micro and macro perspectives to make the proposed hand-in-hand distillation framework more targeted, as shown below:
[0072]
[0073] The added us in the formula represents the weight of each stage from a macro perspective. j represents the microscopic spatial weight of the (i,j) position at the sth stage. In addition, the total feature loss function of each stage will be ∑i,j j is normalized, because different image inputs may result in completely different spatial weight distributions. Comparing this reweighted loss function with the original loss function, we can see that after the vector imitation loss at each position is calculated, it will be multiplied by a micro weight factor, and then the losses at each position will be normalized as the sum of the losses at each stage when accumulated, and then multiplied by a macro weight factor for weighted sum to obtain an improved reweighted imitation loss function. A natural question is how to obtain these weights. Intuitively, it is necessary to evaluate the importance of features from both the overall and partial aspects to calculate the weights. For this reason, the present invention considers some strategies:
[0074] (4.1) Attention weights: The simplest idea is to use a learnable attention module to evaluate the importance of each feature map. However, experimental analysis found that this learnable mechanism will train extremely sparse weights, with a few large weights and most weights being 0. After careful analysis, due to the difficulty of optimizing some imitation losses, the attention module will "lazily" optimize the weights to 0, and the relatively simple weights will become very large, resulting in the final distillation not having a good effect even if some normalization constraints are added.
[0075] (4.2) Focal weight: Inspired by the use of focal loss to mine difficult samples, another attempt in this section is to reduce the weight of the easy-to-imitate part and increase the weight of the difficult-to-imitate part by adding an exponential term: (1-p li )y li , where p li is the loss value l i The proportion of the total value, Y is an adjustable hyperparameter, and experiments show that Y = 2 is more effective. After this reweighting, the loss distribution is more balanced and the distillation is more focused on the difficult part.
[0076] (4.3) Feature mean or variance statistics: Based on the understanding of pruning related research, some network pruning works retain weight parameters with larger absolute values after pruning, because they can play a more critical role in the linear calculation part compared to small weights. Therefore, this paper proposes to calculate the mean statistic or variance statistic of each spatial vector to generate the microscopic spatial weight v:
[0077] v = sigmoid(statistics(v))
[0078] And take the average of all micro weights in each stage and integrate them into the macro weight u of each stage:
[0079] u=mean({v1,v2,…,v n})
[0080] Where statistics represents the mean or variance statistic. It should be noted that using the student characteristic map before transformation to calculate the statistics has better experimental results.
[0081] (4.4) GT-mask: Considering that the detection task focuses more on the key foreground area, the present invention attempts to reduce the influence of irrelevant noise from the background, and uses micro-spatial weights to focus on the foreground area containing the target. Specifically, the micro-spatial weight v=1 is set for the foreground area, and v=0 is used for other areas. For the judgment of the foreground and background areas in the training set pictures, the true value bounding box label can be used to define them. The inside of the bounding box can be regarded as the foreground area, and the outside is regarded as the background area. In addition, the calculation of the macro stage weight "can follow the above-mentioned mean or variance statistic method, and is generated by the micro weight v statistic. The present invention tried to combine these strategies in the experimental stage, and found that the best effect can be achieved by using GT-mask to focus on the foreground area at the micro level and focusing on the stage with large variance statistics at the macro level.
[0082] 5. Distillation training and model deployment prediction stage:
[0083] In distillation training, the distillation optimization objective and the detection optimization objective can be jointly optimized to construct a multi-task loss optimization. The above-mentioned distillation loss is combined with the cross entropy classification loss L commonly used in detection. c and Smooth L1 positioning loss L l Combined, the overall loss is as follows:
[0084] L=L c +λ1L l +λ2L i
[0085] Optimize the parameters of the student network together, where λ1 and λ2 are the weight coefficients of the loss terms, and adjust this coefficient to put all loss terms in the same order of magnitude. During the training process, in order to speed up convergence, the student model will be allowed to self-study for several rounds first, have some detection basis, and then perform distillation training. In addition, the number of iterations is slightly increased in the distillation training to ensure that there are enough training rounds after the introduction of distillation supervision. When the present invention performs model training on the PASCAL VOC dataset and the MS COCO dataset, except for the above-mentioned training differences, other experimental settings and training strategies are the same as the original SSD, including data enhancement, difficult sample mining, and anchor frame setting, without adding additional techniques. When the training is completed and ready for deployment, all additional parts except the original student model will be removed, including the adaptive transformation module added by the student network to facilitate feature matching. After the model is deployed, the target detection stage is performed, and the given image to be detected is input into the trained student model according to a certain preprocessing method. The model will eventually output multiple fixed-size outputs, each of which includes a category label and a position vector, which respectively represent the target position and target category contained in the image.
[0086] Examples:
[0087] In order to prove that the lightweight object detection knowledge distillation method has advantages in performance and adaptability, the present invention verifies and analyzes it through the following experiments:
[0088] A. Experimental Dataset
[0089] Commonly used datasets for general object detection include PASCAL VOC and MS COCO datasets.
[0090] Among them, the PASCAL VOC detection dataset contains 20 categories of natural images: people, birds, cats, cows, dogs, horses, sheep, airplanes, bicycles, ships, buses, cars, motorcycles, trains, bottles, chairs, dining tables, potted plants, plants, sofas and monitors. It is a classic dataset in the detection field. When verifying the model effect, it is necessary to train on the joint dataset of VOC2007trainval and VOC2012trainval, and then test on the VOC 2007test set. The training set has a total of 16,551 pictures, including more than 39,000 target tags, and the test set has 4,952 pictures. The relatively moderate amount of data can quickly verify the model effect. The MS COCO dataset is a more challenging dataset proposed by Microsoft, which mainly includes tasks such as target detection, instance segmentation and key point detection. The 2017 version of the COCO detection dataset is commonly used in the industry. Compared with the PASCAL VOC dataset, the MS COCO dataset contains more common objects in life, totaling 80 categories. The dataset contains a large number of objects in each picture, with a complex background and huge variations in individual sizes. In addition to large objects, it also contains a large number of small objects, making it very difficult. In terms of data volume, the training set totals more than 80,000 images, the validation set 5,000 images, and the test set more than 40,000 images, which need to be specifically verified on the competition website. The overall data volume is very large, and training takes a long time.
[0091] B. Evaluation Criteria
[0092] This paper adopts the mainstream evaluation standards of lightweight object detection research at home and abroad. It is mainly evaluated from three aspects: mAP, computational load (FLOPs) and model parameter quantity:
[0093] For the calculation of mAP accuracy index on the PASCAL VOC dataset, because it is necessary to balance accuracy and recall rate, the PR curve of precision and recall is usually drawn, AP (Average Precision) is calculated, and finally the average of all AP classes is taken to get the mAP on the entire dataset. For the matching judgment between the predicted box and the real box, the IOU intersection and union ratio between the two is greater than the 0.5 threshold as a matching pair, and the confidence threshold is combined to attribute it to TP or FP.
[0094] For the calculation of the mAP accuracy indicator of the MS COCO dataset, in addition to using the intersection-over-union ratio threshold of 0.5, the average accuracy of the intersection-over-union ratio between 0.5 and 0.95 is also calculated. There are also specific evaluation indicators such as precision and recall rate at each scale, and recall rate under a limited number of detection boxes, which are more rigorous.
[0095] For the evaluation index of model computational complexity, this paper uses the number of floating point operations (FLOPs) of a single inference for quantitative analysis, which is also the most commonly used indicator for evaluating model lightweighting at home and abroad.
[0096] The model parameter evaluation index refers to the number of model parameters. It is related to the number of convolutions, the size of the convolution kernel, and the size of the input image. It directly determines the size of the model file and also affects the memory size occupied during model inference.
[0097] C. Experimental Platform
[0098] Hardware: Intel Xeon E5-2630 v4 @ 2.20 GHz, 128 GB memory, NVIDIA GeForce
[0099] TITAN XP×4, video memory 48G.
[0100] Software: operating system Ubuntu 16.04 64-bit, experimental platform PyTorch 1.0.
[0101] D. Experimental results
[0102] (1) Ablation experiment:
[0103] Table 1 Ablation experiments of the proposed hand-in-hand distillation method
[0104]
[0105]
[0106] In order to measure the role of each module in the hand-in-hand distillation framework proposed in this invention and the accuracy gain it brings, the proposed modules are gradually added from the most original distillation, and the change in accuracy is tested. Note the combination of reweighting strategies used here: using GT-mask to focus on the foreground area at the micro level and focusing on the stage with large variance statistics at the macro level achieves the best results. The overall ablation results are shown in Table 1, which shows that the carefully proposed module designs of this method can complement each other, ultimately making the hand-in-hand distillation more effective.
[0107] (2)Comparison with other methods:
[0108] The proposed method is compared with the most advanced object detection distillation methods, including the original HintLearning distillation method, the fine-grained Fine-Grained Distillation, and the Mask Guided distillation method. For fairness, all the comparison methods in the experiment use the same model settings and training settings. The experimental results are shown in Table 2. Through the comparison, it can be seen that the hand-in-hand distillation method proposed in this paper is better than the current best method in all categories, and the overall mAP is 2.4% higher, which makes up for the 63% accuracy gap between the student network and the teacher network, fully proving the effectiveness of this method.
[0109] Table 2 Comparative experiments with other SOTA target detection distillation methods on VOC
[0110]
[0111]
[0112] In order to further test the generalization ability, verification was performed on the challenging MSCOCO dataset, and the training strategy was similar to the previous one. As shown in Table 3, the SSD student model with halved channels achieved a 1.9% mAP (IOU 0.5:0.95) improvement under the proposed hand-in-hand distillation, while the current method only achieved a 0.3% accuracy improvement, which is very little. This is also because the COCO dataset contains smaller and more difficult objects to recognize than VOC, and the teacher network is also very struggling, making it much more difficult to improve accuracy. These experiments fully verify the robustness of the hand-in-hand distillation method proposed in this invention for processing more complex samples.
[0113] Table 3 Distillation comparison experiment on MS COCO dataset
[0114]
[0115] In addition, the present invention also visualizes the network feature graphs before and after distillation. Specifically, this section extracts the Conv9_2 feature graph with a resolution of 5×5 and the Conv10_2 feature graph with a resolution of 3×3 in the student network for visualization analysis. The results are shown in Figure 2. Figure 4 As shown in the figure, the color brightness of each grid represents the activation value strength of the feature at the corresponding position. The closer to yellow, the greater the activation value. By comparing before and after HGD distillation, it can be seen that the key position features containing objects have been significantly enhanced, which naturally leads to better detection results. Some targets that were originally lost in the detection have also been successfully detected.
[0116] The present invention is relatively easy to understand for those skilled in the art. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A lightweight object detection knowledge distillation method, characterized by: 1) The stage of establishing a hand-in-hand distillation framework, including establishing a hand-in-hand distillation framework with multi-level features that fully imitates constraints; 2) Teacher network and student network feature matching stage: including teacher and student model setting, feature map selection and transformation at each stage, and one-to-one feature matching; 3) Multi-stage feature imitation loss measurement stage: Integrate the feature vectors of each stage and each position in the hand-in-hand framework, introduce the cosine distance measurement, and improve the feature imitation loss function; 4) Reweighted Hand-to-Hand Distillation Loss Stage: Construct a more targeted reweighted hand-to-hand distillation imitation loss from both micro and macro perspectives; 5) Distillation training, model deployment and prediction phase.
2. The lightweight object detection knowledge distillation method according to claim 1, characterized in that In the step 1, the distillation framework is established: The distillation method focuses on the problem of high-level features or soft label imitation, adding more shallow feature semantic knowledge extraction as the basis for high-dimensional feature learning. During the distillation training process, the teacher network guides the student network step by step for training optimization.
3. The lightweight object detection knowledge distillation method according to claim 1, characterized in that: In the step 2, the teacher network and the student network feature matching stage: in terms of distillation object setting, the standard one-stage detector SSD based on VGG16 is selected as the teacher network, and the VGG16 SSD model with half the number of channels in each layer is used. It has a similar structure to the teacher network, but only half the number of channels, called channel-half SSD. After preparing the teacher network and the student network, the feature maps of the two networks are matched one by one for subsequent distillation. In the feature extraction stage of the network, each stage dimension of all feature maps is selected and sampled to ensure semantic correspondence. Then use adaptation The transform layer is resized to ensure that the feature map size is consistent: x′ imit =adaption(select(x stage )) Among them, xstage represents the original feature map of each stage, x′imit represents the generated feature map to be matched, and the Adaption transformation layer includes 1x1 convolution, bilinear upsampling and downsampling. Finally, the teacher network and student network feature maps of the same number and size are obtained, which can be matched one-to-one.
4. The lightweight object detection knowledge distillation method according to claim 1, characterized in that: The multi-stage feature imitation loss measurement stage in step 3 is: After establishing the feature matching between the teacher and student networks, the distance loss between the features is calculated to ensure that the student model is optimized towards the teacher model under distillation supervision. In order to solve the problem that the L2 distance optimization between the student feature vector v and the teacher vector Z is too harsh in the multi-stage hand-in-hand distillation framework, the feature imitation loss is improved, and the cosine distance between the student feature vector v and the teacher vector Z at each position is used as the imitation loss function: Combining the feature vectors of each stage and each position, the feature imitation loss is sorted out step by step: Where S is the total number of stages of feature extraction, H s and w s are the height and width of the feature map at each stage, and Represents the student feature vector and teacher feature vector at the (i, j) position in the sth stage respectively. Limit imitation loss uses the cosine metric of Limit_cosine here.
5. The lightweight object detection knowledge distillation method according to claim 1, characterized in that: The re-weighted hand-to-hand distillation loss phase in step 4 is: From both micro and macro perspectives, a re-weighted hand-in-hand distillation imitation loss is introduced to make the distillation more targeted, as shown below: The u added in the formula s Represents the weight of each stage from a macro perspective, Represents the (i, j) of the sth stage In addition, the total feature loss function of each stage will be Normalization, because different image inputs may lead to completely different spatial weight distributions. Compared with the original loss function, we can see that the vector imitation loss at each position will be multiplied by a micro weight factor after calculation, and then the losses at each position will be normalized when accumulated as the sum of the losses at each stage, and then multiplied by a macro weight factor for weighted sum to obtain an improved reweighted imitation loss function.
6. The lightweight object detection knowledge distillation method according to claim 1, characterized in that: In step 5, the distillation training, model deployment and prediction phases are as follows: In distillation training, the above-mentioned imitation distillation loss is combined with the cross entropy classification commonly used in detection Loss L c and Smooth L1 positioning loss L l Combined, the multi-task loss is constructed as follows: L=L c +λ1L l +λ2L i Optimize the parameters of the student network together, where λ1 and λ2 are the weight coefficients of the loss terms. Adjust these coefficients to keep all loss terms at the same order of magnitude. In order to speed up convergence during training, the student model will first be allowed to self-study for a few rounds to have some basic detection, and then distillation training will be performed. In addition, the number of iterations in distillation training is slightly increased to ensure that there are enough training rounds after the introduction of distillation supervision; When training is completed and ready for deployment, all additional parts except the original student model will be removed, including the adaptive transformation module added by the student network to facilitate feature matching. When performing object detection after deployment, the given image to be detected is input into the trained student model according to a certain preprocessing method. The model will eventually output multiple fixed-size outputs, each of which includes a category label and a position vector, representing the target location and target category contained in the image.
Citation Information
Cited By
Copper ore classification method based on lightweight sliding window converter
CN120219862A
Pedestrian re-identification method and device and medium
CN121053706A
