Knowledge distillation-based end-to-end increment target detection method and system

Abstract

The invention discloses an end-to-end increment target detection method and system based on knowledge distillation. The method comprises the following steps: constructing a first task data set and a second task data set; training a teacher model based on the first task data set, training a student model based on the second task data set, calculating loss for classification knowledge and positioning knowledge, and migrating response of the teacher model to the student model by using a knowledge distillation technology to obtain a final student model; and performing target detection based on the final student model. The system comprises a data set construction unit, a distillation unit and a detection unit. By using the method, the detection performance of the old task can be reserved when the new task is learned. The method can be widely applied to the field of high target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to an end-to-end incremental target detection method and system based on knowledge distillation. Background Art

[0002] The human visual and perceptual system is inherently incremental, where new knowledge is learned over time while existing knowledge is preserved. On the other hand, despite their success in object detection, deep neural networks fall short in incremental learning, i.e., when the original model trained on a set of classes is adjusted to additionally detect new classes of objects in the absence of initial training data, they suffer from "catastrophic forgetting" - when a well-trained network adapts to new categories, its performance on old categories drops dramatically. Summary of the invention

[0003] In view of this, in order to solve the technical problem that the existing incremental object detection method cannot retain the old knowledge when adapting to new categories, the present invention proposes an end-to-end incremental object detection method based on knowledge distillation, which includes the following steps:

[0004] Construct old task training datasets and new task training datasets;

[0005] Build a teacher network and train the teacher model based on the old task training dataset to extract the features of the dataset;

[0006] Build a student network and train the student model based on the new task training dataset;

[0007] The losses are calculated for the classification knowledge and the positioning knowledge respectively, and the response of the teacher model is transferred to the student model by using the knowledge distillation technology to obtain the final student model;

[0008] Object detection is performed based on the final student model.

[0009] In some embodiments, the calculation of the loss for the classification knowledge and the positioning knowledge is specifically as follows:

[0010] Using the divide-and-conquer knowledge distillation strategy, the responses of the classification head and regression head are selected as distillation nodes in the head. The SmoothL1 distillation loss is used on the classification knowledge to calculate the probability distribution loss between the teacher model and the student model. The LD localization distillation loss is used on the localization knowledge to calculate the probability distribution loss between the teacher model and the student model, so as to transfer the knowledge of the old task from the teacher network to the student network.

[0011] In some embodiments, it also includes:

[0012] Integrate the deformable convolution module and the efficient pyramid attention segmentation module into the backbone network of the teacher network to improve the accuracy of feature extraction, so that it can better match the shape and size of the target object. Redesign the high-performance HRNetV2p neck module to better integrate and transfer multi-scale features.

[0013] In order to enable the student model to better learn the old knowledge of the teacher model, the deformable convolution module and the efficient pyramid attention segmentation module are also integrated into the backbone network of the student network, and the neck module of HRNetV2p is also used.

[0014] The present invention also proposes an end-to-end incremental target detection system based on knowledge distillation, the system comprising:

[0015] A data set construction unit, used to construct a first task data set and a second task data set;

[0016] A distillation unit, which trains a teacher model based on the first task data set, trains a student model based on the second task data set, calculates losses for classification knowledge and positioning knowledge respectively, and uses knowledge distillation technology to transfer the response of the teacher model to the student model to obtain a final student model;

[0017] A detection unit performs target detection based on the final student model.

[0018] Based on the above scheme, the present invention provides an end-to-end incremental target detection method and system based on knowledge distillation. Considering the catastrophic forgetting problem of the network, knowledge technology is used to transfer the knowledge of the teacher model to the student model, thereby improving the learning efficiency and model generalization ability of the network. Considering the different distributions of different knowledge, it is not necessarily beneficial to transfer classification knowledge and positioning knowledge at the same location at the same time. This patent adopts a divide-and-conquer strategy, in which different distillation methods are used for classification knowledge and positioning knowledge, which greatly improves the knowledge transfer performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of the steps of an end-to-end incremental target detection method based on knowledge distillation of the present invention;

[0020] Figure 2 It is a schematic diagram of the overall framework of the method of the present invention;

[0021] Figure 3 is a schematic diagram of the knowledge distillation process of a specific embodiment of the present invention;

[0022] Figure 4 Schematic diagram of the data processing process of the deformable convolution module in a specific embodiment of the present invention;

[0023] Figure 5It is a schematic diagram of the data processing process of the efficient pyramid attention segmentation module in a specific embodiment of the present invention;

[0024] Figure 6 It is a schematic diagram of the data processing process of the neck module of HRNetV2p in a specific embodiment of the present invention;

[0025] Figure 7 Schematic diagram of the data processing process of LD positioning distillation in a specific embodiment of the present invention;

[0026] Figure 8 It is a structural block diagram of an end-to-end incremental target detection system based on knowledge distillation of the present invention. DETAILED DESCRIPTION

[0027] When state-of-the-art traditional object detectors are fine-tuned for new tasks, they often fail on previously trained tasks, a problem known as catastrophic forgetting. Incremental learning can effectively solve this problem. The end-to-end incremental object detection algorithm based on response knowledge distillation in the present invention first trains the old task on the teacher network detector, then uses a divide-and-conquer knowledge distillation strategy to select the responses of the classification head and regression head as distillation nodes in the head, and uses the Smooth L1 distillation loss on the classification knowledge to calculate the probability distribution loss between the teacher network and the student network, and uses the LD localization distillation loss on the localization knowledge to calculate the probability distribution loss between the teacher network and the student network, so as to transfer the knowledge of the old task from the teacher network to the student network. At this time, the student network detector has the ability to recognize new knowledge while not forgetting the old knowledge.

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0030] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0031] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0032] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0033] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0034] Reference Figure 1 , which is a flow chart of an optional example of an end-to-end incremental target detection method based on knowledge distillation proposed in the present invention. The method can be applied to a computer device. The target detection method proposed in this embodiment may include but is not limited to the following steps:

[0035] Step S1, constructing a first task data set and a second task data set;

[0036] Step S2: training a teacher model based on the first task data set, training a student model based on the second task data set, calculating losses for classification knowledge and positioning knowledge respectively, and using knowledge distillation technology to transfer the response of the teacher model to the student model to obtain a final student model;

[0037] Among them, the Smooth L1 distillation loss is used on the classification knowledge to calculate the probability distribution loss between the teacher model and the student model; the LD positioning distillation loss is used on the positioning knowledge to calculate the probability distribution loss between the teacher model and the student model.

[0038] For the incremental learning task, a loss function is designed to calculate the classification and localization loss L of the student detector model, considering the differences in classification knowledge and positioning knowledge of the teacher network, the loss λ of the teacher network and the student network in the category are calculated respectively cls L cls (C T ,C S ) and localization loss λ bbox L bbox (C T ,C S ), ,λ bbox ,λ cls It is a hyperparameter that enables it to autonomously select the best value. The weighted sum of the three losses constitutes the target loss function of the entire network, ensuring the incremental learning effect and final performance of the model.

[0039] Step S3: performing target detection based on the final student model.

[0040] The overall framework of this specific embodiment is as follows Figure 2 shown.

[0041] The core content of knowledge distillation technology is: once the complex model is trained, a different training method called "distillation" is used to transfer knowledge from the complex model (called the teacher model) to a smaller model (called the student model) that is easier to deploy. Figure 3 In the example, the network model outputs n logits z, which are transformed by a softmax with temperature τ to obtain a probability distribution p = S (z, τ). Then, we only need to minimize the student probability distribution p S The probability of having the teacher is p T By using the KL divergence loss, the knowledge of the large model can be transferred to the small model.

[0042] In some feasible embodiments, it also includes:

[0043] A deformable convolution module, an efficient pyramid attention segmentation module and a neck module of HRNetV2p are introduced into the basic network to obtain a teacher model; the student model has the same network structure as the teacher model.

[0044] The processing process of the deformable convolution module is as follows: Figure 4 The deformable convolution module considers that different stages, different feature maps, and even different pixels may have their optimal convolution kernel structures. Therefore, the convolution kernel should not be a simple rectangle, but an offset should be learned based on each ordinary convolution kernel. Figure 3In the example above, it acts on the input feature map through a Conv operation to obtain a set of prediction results of the convolution kernel offset. The size of this offset feature map remains the same as the input feature map. The number of channels of the offset feature map is 2N, where 2 means that each offset is two values ​​(x, y), and N is the number of pixels in the convolution kernel. For example, when the size of the convolution kernel is 4×4, N=16. From this, it can be seen that the deformable convolution learns a set of offsets for each pixel of the input feature map, and different numbers of channels of the same feature map use the same prediction offset.

[0045] The processing process of the efficient pyramid attention segmentation module is referenced Figure 5 , embedding the attention module into the existing CNN can bring significant performance improvements. However, there are still two challenging problems to be solved. One is how to effectively obtain and utilize the spatial information of feature maps of different scales to enrich the feature space. Second, channel attention or spatial attention can only effectively capture local information, but cannot establish long-term dependencies. Based on channel attention, this patent introduces the multi-scale idea and proposes a pyramid segmentation attention module, namely PSA module. Figure 5 In the proposed method, the SPC module is first used to segment the channels, and then multi-scale features are extracted for the spatial information on each channel feature map; secondly, the SEWeight module is used to extract the channel attention of feature maps of different scales to obtain the channel attention vector at each different scale; then Softmax is used to recalibrate the multi-scale channel attention vector to obtain the new attention weight after multi-scale channel interaction; finally, the recalibrated weights and the corresponding feature maps are element-wise multiplied to obtain a feature map after multi-scale feature information attention weighting, which has richer multi-scale information representation capabilities.

[0046] For the input feature map X, first divide the feature map X into S parts, using [X0,X1,...X S-1 ] to indicate that the number of channels in each segmented part is At the same time, each feature map X after segmentation i ∈R C′×H×W ,i=0,1,...,S-1Therefore, for each divided channel feature map, we apply multi-scale convolution to extract the spatial information of feature maps of different scales. Here we mainly use multi-scale convolution kernel group convolution. The role of group convolution is mainly to reduce the amount of parameters.

[0047] The processing process of the neck module of HRNetV2p refers to Figure 6, HRNetV2p connects high-resolution and low-resolution images in parallel, exchanges information across resolutions, and forms a feature pyramid in the output stream for object detection. Figure 6 In the method, all resolution feature maps (small feature maps upsample) are first concatenated and feature maps of different resolutions are connected in parallel. High-resolution feature maps are always maintained throughout the network process, which is very helpful in capturing detailed information. Then a feature pyramid network is used to fuse low-resolution feature maps with strong language information and high-resolution feature maps with weak semantic information but rich spatial information with less computational effort. This allows the network to better model and locate different positions and scales of input targets, effectively enhancing target detection performance.

[0048] In some feasible embodiments, the data processing process of LD positioning distillation refers to Figure 7 , the bounding box bboxes of target detection can be expressed as e∈β={t,b,l,r}, where tblr represents the distance from the point to the four edges of up, down, left and right respectively. S(.,τ) is a generalized softMax function with temperature τ. For a given detector, we first convert the bounding box representation into a probability distribution. Then, the distillation position is determined by region weighting through the main distillation area and the valuable localization area. Finally, the positioning loss between the two probability distributions predicted by the teacher model and the student model is calculated.

[0049] Figure 7 Region Weighting structure in. In the multi-task learning framework, the feature map is equivalent to the root of the tree, and the downstream heads are equivalent to the leaves of the tree. Then the feature map obviously contains the knowledge required by all leaves. When performing feature distillation, classification knowledge and positioning knowledge will naturally be transferred. However, due to the different distribution of the two kinds of knowledge, it is naturally not beneficial to transfer both classification knowledge and positioning knowledge at the same location. It is very likely that some areas are only beneficial to the transfer of classification knowledge, and it is also possible that some areas are only beneficial to the transfer of positioning knowledge. In other words, we need to divide and conquer and transfer knowledge according to local conditions. This is obviously something that feature distillation is powerless to do, because it only transfers mixed knowledge. This patent uses multi-task learning to naturally decouple knowledge into different types, which allows knowledge distillation to be performed selectively in a region. To this end, this specific embodiment introduces the concept of a valuable positioning region VLR (ValuableLocalization Region) to help us perform divide and conquer distillation.

[0050] like Figure 8 As shown, an end-to-end incremental object detection system based on knowledge distillation includes:

[0051] A data set construction unit, used to construct a first task data set and a second task data set;

[0052] A distillation unit, which trains a teacher model based on the first task data set, trains a student model based on the second task data set, calculates losses for classification knowledge and positioning knowledge respectively, and uses knowledge distillation technology to transfer the response of the teacher model to the student model to obtain a final student model;

[0053] A detection unit performs target detection based on the final student model.

[0054] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0055] An end-to-end incremental object detection device based on knowledge distillation:

[0056] at least one processor;

[0057] at least one memory for storing at least one program;

[0058] When the at least one program is executed by the at least one processor, the at least one processor implements the end-to-end incremental target detection method based on knowledge distillation as described above.

[0059] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0060] A storage medium stores processor-executable instructions, which, when executed by a processor, are used to implement an end-to-end incremental target detection method based on knowledge distillation as described above.

[0061] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0062] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An end-to-end incremental object detection method based on knowledge distillation, characterized in that: The following steps are involved: Construct a first task data set and a second task data set; Training a teacher model based on the first task data set, training a student model based on the second task data set, calculating losses for classification knowledge and positioning knowledge respectively, and using knowledge distillation technology to transfer the response of the teacher model to the student model to obtain a final student model; Object detection is performed based on the final student model.

2. According to claim 1, the end-to-end incremental target detection method based on knowledge distillation is characterized in that: The specific calculation of the loss for classification knowledge and positioning knowledge is as follows: Using Smooth L1 distillation loss on classification knowledge to calculate the probability distribution loss between the teacher model and the student model; The LD localization distillation loss is used on the localization knowledge to calculate the probability distribution loss between the teacher model and the student model.

3. According to claim 1, the end-to-end incremental target detection method based on knowledge distillation is characterized in that: Also includes: The deformable convolution module, efficient pyramid attention segmentation module and neck module of HRNetV2p are introduced into the basic network to obtain the teacher model.

4. According to claim 3, the end-to-end incremental target detection method based on knowledge distillation is characterized in that: The student model has the same network structure as the teacher model.

5. According to claim 3, the end-to-end incremental target detection method based on knowledge distillation is characterized in that: In the deformable convolution module, the working process is as follows: Perform a convolution operation on the input feature map to obtain an offset feature map.

6. According to claim 3, the end-to-end incremental object detection method based on knowledge distillation is characterized in that: In the efficient pyramid attention segmentation module, the working process is as follows: Split the input feature map to obtain a channel feature map; Performing multi-scale feature extraction on the channel feature map, and extracting channel attention to obtain a channel attention vector; Recalibrate the channel attention vector using Softmax to obtain a calibrated weight; A dot multiplication operation is performed on the calibrated weight and the corresponding feature map to obtain a second feature map.

7. The end-to-end incremental object detection method based on knowledge distillation according to claim 3, characterized in that: In the neck module of the HRNetV2p, the working process is as follows: The input feature maps are spliced ​​according to the resolution; Based on the feature pyramid, the low-resolution feature map and the high-resolution feature are fused to obtain the third feature map.

8. An end-to-end incremental object detection system based on knowledge distillation, characterized in that: include: A data set construction unit, used to construct a first task data set and a second task data set; A distillation unit, which trains a teacher model based on the first task data set, trains a student model based on the second task data set, calculates losses for classification knowledge and positioning knowledge respectively, and uses knowledge distillation technology to transfer the response of the teacher model to the student model to obtain a final student model; A detection unit performs target detection based on the final student model.

9. An end-to-end incremental object detection device based on knowledge distillation, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements an end-to-end incremental target detection method based on knowledge distillation as described in any one of claims 1-7.

Images