Incremental target detection method, apparatus, equipment and media based on fine response distillation

Abstract

This invention discloses an incremental target detection method, apparatus, device, and medium based on fine-grained response distillation. This method, while the target detector learns new target categories end-to-end, maintains the detector's detection capability for previously learned target categories by selecting fine-grained knowledge distillation regions and applying different knowledge distillation strategies to different regions. This allows target detectors, originally only applicable to fixed categories, to adapt to situations where the number of detectable categories is constantly increasing, improving the target detector's continuous learning capability and better adapting to dynamically changing real-world scenarios. It enhances the detector's ability to retain knowledge of existing categories when learning new categories, improving the continuous learning capability of conventional target detectors in real-world dynamic scenarios. Simultaneously, it expands the adaptability of conventional detectors to dynamically growing detectable categories, broadening the application scenarios of target detectors, improving the efficiency of incremental learning, and enabling further application to incremental target detection in specific business scenarios.

Description

Technical Field

[0001] This invention relates to the fields of target detection and incremental learning technology, and in particular to an incremental target detection method, apparatus, equipment and medium based on fine response distillation. Background Technology

[0002] Object detection involves classifying and locating targets of interest in images or videos. The current mainstream approach is based on object detectors using deep neural networks. Existing object detectors are often designed for detection datasets with fixed target categories. Once the detection model is built, the types of targets it can detect remain unchanged and cannot be incrementally expanded to meet actual business needs. If it is necessary to detect more target categories, a new object detection dataset containing both existing and new categories must be constructed, and the detector must be trained from scratch.

[0003] However, in real-world scenarios such as real-time online detection systems, data of different categories often appears in a "streaming" manner, placing higher real-time demands on the incremental expansion of the categories detectable by the object detector. Since a dataset containing all categories needs to be built each time, learning the object detector from scratch is inefficient and cannot meet the needs of real-world scenarios. Furthermore, in real-world scenarios, data privacy protection often prevents access to data from previous stages from being accessible during incremental learning, making the traditional approach of building a detector capable of recognizing more object categories from scratch impractical. Summary of the Invention

[0004] In view of the above problems, this invention provides an incremental target detection method, apparatus, device, and medium based on fine-grained response distillation to overcome or at least partially solve the above problems. Addressing the issue of poor scalability of common target detectors as the number of target categories increases, this invention proposes an incremental target detection method based on fine-grained response knowledge distillation, using the generalized FocalLoss (GFL) detector. While the target detector learns new target categories end-to-end, it maintains its detection capability for previously learned target categories by selecting fine-grained knowledge distillation regions and applying different knowledge distillation strategies to different regions. This allows target detectors, originally only applicable to fixed categories, to adapt to the continuously increasing number of target categories, improving the continuous learning capability of the target detector and better adapting to dynamically changing real-world scenarios.

[0005] This invention provides the following solution:

[0006] An incremental target detection method based on fine response distillation includes:

[0007] The target detector that acquires knowledge about distillation is used as the teacher detector; the teacher detector is a detector that has been learned on the original category data;

[0008] The network structure of the teacher detector is copied, and the number of output channels of its classification branch is expanded to form a student detector that needs to learn new categories while retaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned.

[0009] Training data is obtained by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification and localization responses based on the training data, respectively; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and segmented as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector;

[0010] The classification score with the highest confidence in the teacher detector's classification response is used as the distillation value quality index of the classification branch. Candidate classification distillation regions are divided into high-value and low-value classification distillation regions through bimodal Gaussian mixture distribution modeling. The joint representation of the distillation value index of the teacher detector's classification branch and the information entropy of the localization branch response distribution is used as the distillation value quality index of the localization branch. Candidate localization distillation regions are divided into high-value and low-value localization distillation regions through bimodal Gaussian mixture distribution modeling.

[0011] In the high-value classification distillation region, the classification response of the teacher detector is decoupled into a maximum class response and a non-maximum class response. The student detector is then subjected to a maximum response class knowledge distillation method and a non-maximum response class knowledge distillation method to obtain knowledge from the high-value classification response of the teacher detector. In the low-value classification distillation region, the difference in response between the student detector and the teacher detector is transferred using an L1 norm knowledge distillation method.

[0012] In the high-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a first distillation temperature to obtain location knowledge for the high-value region; in the low-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a second distillation temperature to obtain location knowledge for the low-value region; the first distillation temperature is greater than the second distillation temperature.

[0013] The student detector is iteratively trained using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

[0014] Preferably: the distillation region is finely divided from the response of the teacher detector, including a high-value classification distillation region, a low-value classification distillation region, a high-value location distillation region, and a low-value location distillation region; including:

[0015] The threshold segmentation of the classification confidence of the teacher detector is used as a coarse selection scheme for valuable distillation regions, and candidate distillable regions corresponding to the classification response and localization response of the teacher detector are selected.

[0016] High-value distillation regions and low-value distillation regions are obtained by extracting from the candidate distillable regions.

[0017] Preferably: the distillation value quality index Q of the classification branch cls It can be expressed by the following formula:

[0018]

[0019] In the formula: C T Let σ be the classification response of the teacher detector, and σ be the Sigmoid activation function.

[0020] Preferably: the distillation value quality index Q of the positioning branch loc It can be expressed by the following formula:

[0021]

[0022]

[0023] In the formula: E(p) is the general distribution information entropy, M is the category length corresponding to each distribution, and G is the predicted localization response distribution on each feature node.

[0024] Preferably: the distillation value quality index of the aforementioned classification branch Threshold segmentation is performed to obtain the same candidate distillation regions in both the classification and localization responses;

[0025] Based on the candidate regions and A dual-modal Gaussian mixture distribution is used for modeling, and the optimal boundary value between the two Gaussian distributions after modeling is used as the dynamic threshold for refining the candidate region segmentation.

[0026] Regions with quality values ​​greater than the dynamic threshold are designated as high-value distillation regions, while regions with quality values ​​lower than the dynamic threshold are designated as low-value distillation regions.

[0027] Preferably: the maximum category response p in the maximum response class knowledge distillation and the non-maximum response class knowledge distillation max Non-maximum category response p not_max They are respectively expressed by the following formulas:

[0028]

[0029]

[0030] In the formula: max(Z) and max_id(Z) represent obtaining the maximum component and the index of the maximum component of Z respectively, and N is the total number of categories. This represents the response output of the high-value region classification branch.

[0031] Preferably: the overall loss function L of knowledge distillation distill It can be expressed by the following formula:

[0032]

[0033] In the formula: The distillation loss function represents the distillation loss function for high-value classification distillation regions. The distillation loss function represents the distillation loss function for low-value classification distillation regions. This represents the distillation loss function for high-value positioning distillation regions. This represents the distillation loss function for low-value distillation regions.

[0034] An incremental target detection device based on fine response distillation includes:

[0035] The teacher detector acquisition unit is used to acquire the target detector of the knowledge to be distilled as the teacher detector; the teacher detector is a detector that has been learned on the original category data;

[0036] The student detector acquisition unit is used to copy the network structure of the teacher detector and expand the number of output channels of its classification branch to form a student detector that needs to learn new categories while retaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned.

[0037] The candidate distillation region determination unit is used to obtain training data by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification and localization responses respectively based on the training data; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and used as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector;

[0038] The refined distillation region determination unit is used to divide candidate classification distillation regions into high-value and low-value classification distillation regions by using the classification score with the highest confidence in the teacher detector classification response as the distillation value quality index of the classification branch and by modeling with a bimodal Gaussian mixture distribution; and by using the joint representation of the distillation value index of the teacher detector classification branch and the information entropy of the localization branch response distribution as the distillation value quality index of the localization branch and by modeling with a bimodal Gaussian mixture distribution, candidate localization distillation regions are divided into high-value localization distillation regions and low-value localization distillation regions.

[0039] The classification branch knowledge distillation unit is used to decouple the classification response of the teacher detector into a maximum class response and a non-maximum class response in the high-value classification distillation region, and to extract knowledge from the high-value classification response of the teacher detector by using the maximum response class knowledge distillation method and the non-maximum response class knowledge distillation method respectively for the student detector; in the low-value classification distillation region, knowledge is transferred by using the L1 norm knowledge distillation method for the response difference between the student detector and the teacher detector.

[0040] A localization branch knowledge distillation unit is used to obtain localization knowledge of the high-value region by using localization knowledge distillation at a first distillation temperature on the responses of the student detector and the teacher detector in the high-value localization distillation region; and to obtain localization knowledge of the low-value region by using localization knowledge distillation at a second distillation temperature on the responses of the student detector and the teacher detector in the low-value localization distillation region; wherein the first distillation temperature is greater than the second distillation temperature.

[0041] The student detector training unit is used to iteratively train the student detector using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

[0042] An incremental target detection device based on fine response distillation, the device comprising a processor and a memory:

[0043] The memory is used to store program code and transmit the program code to the processor;

[0044] The processor is used to execute the above-described incremental target detection method based on fine response distillation according to the instructions in the program code.

[0045] A computer-readable storage medium for storing program code for executing the above-described incremental target detection method based on fine response distillation.

[0046] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0047] This application provides an incremental target detection method, apparatus, device, and medium based on fine-grained response distillation. This method, while the target detector learns new target categories end-to-end, maintains the detector's detection capability for previously learned target categories by selecting fine-grained knowledge distillation regions and applying different knowledge distillation strategies to different regions. This allows target detectors, originally only applicable to fixed categories, to adapt to situations where the number of detectable categories continuously increases, improving the detector's continuous learning capability and better adapting to dynamically changing real-world scenarios. It enhances the detector's ability to retain knowledge of existing categories when learning new categories, improving the continuous learning capability of conventional target detectors in real-world dynamic scenarios. Simultaneously, it expands the adaptability of conventional detectors to dynamically increasing detectable categories, broadening the application scenarios of target detectors, improving the efficiency of incremental learning, and enabling further application to incremental target detection in specific business scenarios.

[0048] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0050] Figure 1 This is a flowchart of the incremental target detection method based on fine response distillation provided in this embodiment of the invention;

[0051] Figure 2 This is a framework diagram of the incremental target detection system based on refined response distillation provided in this embodiment of the invention;

[0052] Figure 3 This is a structural diagram of the refined response knowledge distillation functional module provided in an embodiment of the present invention;

[0053] Figure 4 This is a flowchart of the dynamic distillation region division module provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the refined distillation strategy for different response regions provided in the embodiments of the present invention;

[0055] Figure 6 This is a schematic diagram of the incremental target detection device based on fine response distillation provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of an incremental target detection device based on fine response distillation provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0058] See Figure 1 This invention provides an incremental target detection method based on fine response distillation, as exemplified by the present invention. Figure 1 As shown, the method may include:

[0059] S101: The target detector that acquires knowledge about distillation is used as the teacher detector; the teacher detector is a detector that has been learned on the original category data;

[0060] S102: Copy the network structure of the teacher detector and expand the number of output channels of its classification branch to form a student detector that needs to learn new categories while maintaining the knowledge of the original categories; the localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned.

[0061] In practice, when the detector built on a dataset containing A target categories is incrementally adapted to B new target categories, the network structure of the detector first needs to be modified to adapt it to (A+B) target categories.

[0062] Common object detectors typically include classification and localization branches. The classification branch predicts the classification probabilities of different classes on the downsampled feature map, while the localization branch predicts the localization information (such as rectangular bounding boxes) of the target of interest on the feature map. The output response of the classification branch is related to the target classes that the detector can detect, while the prediction response of the localization branch is independent of the target class. Therefore, when incrementally expanding the network structure, the classification branch needs to be expanded so that the number of output classes changes from A to A+B.

[0063] The detector built on the existing target category dataset A is called the teacher detector, and the detector prepared to incrementally learn B new target categories is called the student detector.

[0064] S103: Training data is obtained by sampling from the new target category data, so that the teacher detector and the student detector generate corresponding classification and localization responses respectively based on the training data; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and segmented as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector;

[0065] In specific implementation, the threshold segmentation of the classification confidence of the teacher detector is used as a coarse selection scheme for valuable distillation regions, and candidate distillable regions for obtaining the classification response and localization response of the teacher detector are selected.

[0066] In practical applications, it is necessary to select appropriate distillation regions based on the teacher detector's response. Distilling in valuable regions helps to better retain knowledge gained from the original task during the learning process of the new task. In real incremental object detection scenarios, the original category labels are not visible during the learning process of the new task, making it difficult for the new detector model to clearly distinguish which regions in the response to the original category are valuable and which are not when learning the new category. In mainstream object detectors, removing background information with low confidence is a necessary step in the post-processing of prediction results. A common practice is to threshold the confidence of the classification prediction results.

[0067] Inspired by this, the embodiments of this application use threshold segmentation based on classification confidence as a coarse selection of valuable distillation regions, dividing a sufficiently large distillable region as a candidate distillable region.

[0068] S104: Using the classification score with the highest confidence in the teacher detector's classification response as the distillation value quality index of the classification branch, the candidate classification distillation region is divided into high-value classification distillation region and low-value classification distillation region through bimodal Gaussian mixture distribution modeling; using the joint representation of the distillation value index of the teacher detector's classification branch and the information entropy of the localization branch response distribution as the distillation value quality index of the localization branch, the candidate localization distillation region is divided into high-value localization distillation region and low-value localization distillation region through bimodal Gaussian mixture distribution modeling;

[0069] In specific implementation, the distillation value quality index Q of the classification branch cls It can be expressed by the following formula:

[0070]

[0071] In the formula: C T Let σ be the classification response of the teacher detector, and σ be the Sigmoid activation function.

[0072] The distillation value quality index Q of the positioning branch loc It can be expressed by the following formula:

[0073]

[0074]

[0075] In the formula: E(p) is the general distribution information entropy, M is the category length corresponding to each distribution, and G is the predicted localization response distribution on each feature node.

[0076] The distillation value quality index Q for the aforementioned classification branch cls Threshold segmentation is performed to obtain the same candidate distillation regions in both the classification and localization responses;

[0077] Based on the Q of the candidate regions respectively cls and Q loc A dual-modal Gaussian mixture distribution is used for modeling, and the optimal boundary value between the two Gaussian distributions after modeling is used as the dynamic threshold for refining the candidate region segmentation.

[0078] Regions with quality values ​​greater than the dynamic threshold are designated as high-value distillation regions, while regions with quality values ​​lower than the dynamic threshold are designated as low-value distillation regions.

[0079] This invention divides the classification response and localization response into refined distillation regions, namely, high-value and low-value regions.

[0080] First, distillation value quality indices are constructed for the classification and localization branches respectively. The distillation quality index for the classification branch is the classification score with the highest confidence level. The distillation quality for the localization branch is related not only to the classification confidence level (too low a confidence level will result in rejection and prevent optimization) but also to the information entropy of the distribution corresponding to that node. The higher the maximum information entropy of the distribution, the higher its distillable knowledge value. Therefore, the joint representation of the classification score and the information entropy of the localization response distribution is selected as the distillation value quality index for the localization branch.

[0081] For the classification distillation quality index and the localization distillation quality index of the candidate region, two modes of clustering are performed by modeling with a bimodal Gaussian mixture distribution. The candidate regions of the classification branch and the localization branch are dynamically divided into high-value distillation regions and low-value distillation regions, respectively, to guide the formulation of more detailed strategies in the process of classification and localization knowledge transfer.

[0082] S105: In the high-value classification distillation region, the classification response of the teacher detector is decoupled into the maximum class response and the non-maximum class response. The maximum response class knowledge distillation method and the non-maximum response class knowledge distillation method are used to obtain knowledge from the high-value classification response of the teacher detector for the student detector. In the low-value classification distillation region, the L1 norm knowledge distillation method is used to transfer knowledge about the difference between the responses of the student detector and the teacher detector.

[0083] In specific implementation, the maximum class response p in the maximum response class knowledge distillation and the non-maximum response class knowledge distillation max Non-maximum category response p not_max They are respectively expressed by the following formulas:

[0084]

[0085]

[0086] In the formula: max(Z) and max_id(Z) represent obtaining the maximum component and the index of the maximum component of Z respectively, and N is the total number of categories. This represents the response output of the high-value region classification branch.

[0087] In practical applications, the method provided in this application embodiment performs fine knowledge distillation on the high-value distillation region and low-value distillation region of the classification branch respectively.

[0088] For the classification branch, the teacher detector scores high in high-value regions, indicating a greater likelihood of target presence and thus more prominent knowledge about the target of interest. We decouple high-value regions into maximum response class knowledge distillation and non-maximum response class knowledge distillation. This allows the importance of the maximum and non-maximum target classes to be independently tuned, promoting better learning of target object knowledge and improving the flexibility and effectiveness of distillation. Low-value regions in the classification branch score lower and are more likely to be background in the image to be detected. Knowledge distillation of background regions is equally important during incremental learning. To better preserve this background-biased knowledge, the method provided in this application uses the L1 norm of the difference in responses between the new category data and the teacher and student detection models as a fast optimization objective for knowledge distillation.

[0089] S106: In the high-value positioning distillation region, the responses of the student detector and the teacher detector are distilled using positioning knowledge at a first distillation temperature to obtain positioning knowledge of the high-value region; in the low-value positioning distillation region, the responses of the student detector and the teacher detector are distilled using positioning knowledge at a second distillation temperature to obtain positioning knowledge of the low-value region; the first distillation temperature is greater than the second distillation temperature.

[0090] Because the localization branch optimizes more extensively in areas with a higher probability of being positive samples during detector training, the localization results in high-value regions are more reliable and more likely to contain the target. Low-value regions are optimized less and are more susceptible to noise interference.

[0091] Therefore, the solution provided in this application employs different distillation temperatures for location knowledge transfer in high- and low-value areas to fully leverage the transfer of location knowledge. Higher distillation temperatures are used in high-value areas to ensure that different response values ​​in the location response distribution are considered. Slightly lower distillation temperatures are used in low-value areas, allowing the student detector to focus more on the higher values ​​in the low-value area response distribution, thereby avoiding noise interference from low response values.

[0092] The overall loss function L of knowledge distillation distill It can be expressed by the following formula:

[0093]

[0094] In the formula: The distillation loss function represents the distillation loss function for high-value classification distillation regions. The distillation loss function represents the distillation loss function for low-value classification distillation regions. This represents the distillation loss function for high-value positioning distillation regions. This represents the distillation loss function for low-value distillation regions.

[0095] S107: The student detector is iteratively trained using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

[0096] The incremental target detection method based on fine-grained response distillation provided in this application maintains the detector's ability to detect previously learned target categories while the target detector learns new target categories end-to-end. This is achieved by selecting fine-grained knowledge distillation regions and applying different knowledge distillation strategies to different regions. This allows a target detector that was originally only applicable to fixed categories to adapt to situations where the number of categories to be detected is constantly increasing, improving the target detector's continuous learning ability and better adapting to dynamically changing real-world scenarios.

[0097] The incremental target detection method and system based on refined knowledge distillation response comprises three parts: incremental expansion of the detector network structure, determination of the knowledge distillation region, and application of refined knowledge distillation strategies to different regions. To accommodate the increasing number of target categories, the detector network structure needs to be expanded accordingly to adapt to the detection capabilities for more target categories.

[0098] Meanwhile, selecting more precise distillation regions and applying more refined knowledge distillation strategies can help retain the knowledge learned on the original categories better, enabling the target detector to not only learn knowledge of new target categories but also maintain the detection capability of the original categories, thereby incrementally expanding the detectable target categories.

[0099] The method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0100] See Figure 2 Let Task 1, Task 2, ..., Task N represent tasks with progressively increasing categories. For Task 1, Detector 1 (the teacher detector) performs end-to-end learning based on the input data D1, ultimately enabling the detector to detect A target categories. Task 2 requires the detector to learn only data D2, where the target categories labeled in D2 do not overlap with those labeled in data D1. In the incremental learning of Task 2, Detector 2 (the student detector) first expands its network structure so that its output response can represent A+B target categories in terms of dimensions.

[0101] The specific expansion method involves increasing the number of output channels in the last layer of the detector classification network, changing its channel count from A to A+B. Simultaneously, the network parameters trained by detector 1 are loaded into detector 2 as initial network parameters. Then, detector 2 learns end-to-end from the labeled data D2 to learn how to detect targets in the target class set C2.

[0102] Simultaneously, the proposed knowledge distillation method is used to transfer detection capabilities for the category set C1 from detector 1. Finally, detector 2 achieves the ability to simultaneously detect A+B target categories. As tasks 3, 4, ..., N arrive, incremental learning is performed in the same way, thereby achieving incremental target detection for scenarios with an ever-increasing number of categories.

[0103] See Figure 3 The refined response knowledge distillation mainly includes a distillation region partitioning module and a refined knowledge distillation module. During the learning process of detector 2, data sampled from data D2 is simultaneously fed into detector 1 and detector 2, generating a classification response C on detector 1. T and location response B T A classification response C is generated on detector 2. S1 Classification response C S2 and positioning response B S .

[0104] The distillation region segmentation module uses the classification response C from detector 1. T and location response B T The fine regions of classification distillation and location distillation are determined in the middle.

[0105] Next, the classification responses C from detector 1 and detector 2 are analyzed. T and C S1 Simultaneously, the response values ​​corresponding to the fine-grained regions of classification distillation are extracted, and a refined classification knowledge distillation strategy is applied.

[0106] The location response B from detector 1 and detector 2 T and B S Simultaneously, the response values ​​corresponding to the fine-grained positioning distillation region are extracted, and a refined positioning knowledge distillation strategy is applied.

[0107] Through this series of operations, the detection capability for category set C1 was transferred.

[0108] See Figure 4 First, define the distillation quality index Q for the classification branch and the positioning branch respectively. cls and Q loc .

[0109] Classification of branch distillation quality index Q cls It is taken directly from the maximum value of the classified response, defined as follows:

[0110]

[0111] Among them, C TLet σ be the classification response of the teacher detector, and σ be the Sigmoid activation function.

[0112] Quality index Q of branch distillation loc Using the distillation mass Q of the branch distillation cls The information entropy of the location response distribution is jointly represented and defined as follows:

[0113]

[0114]

[0115] Where E(p) is the general distribution information entropy, M is the category length corresponding to each distribution, and G is the predicted localization response distribution on each feature node.

[0116] Quality index Q for branch distillation cls Threshold segmentation is performed to obtain candidate distillation regions that are identical in both classification and localization responses. Then, based on the Q of each candidate region... cls and Q loc A bimodal Gaussian mixture distribution is used for modeling. The optimal boundary value between the two Gaussian distributions after modeling is used as the dynamic threshold for refining the candidate region. Regions with quality values ​​greater than the dynamic threshold are designated as high-value distillation regions, while regions with quality values ​​lower than the dynamic threshold are designated as low-value distillation regions.

[0117] Finally, the candidate regions for classification branches are dynamically divided into high-value classification distillation regions and low-value classification distillation regions, and the candidate regions for positioning branches are dynamically divided into high-value positioning distillation regions and low-value positioning distillation regions.

[0118] See Figure 5 For high-value regions of classification branches, the probability of a target is greater. Therefore, in this embodiment, the high-value classification distillation region is decoupled into maximum response class knowledge distillation and non-maximum response class knowledge distillation to reduce the mutual inhibition caused by the coupling between the maximum response class and the non-maximum response class.

[0119] use The response of the high-value region classification branch is represented by the maximum class response p. max Non-maximum category response p not_max They are defined as follows:

[0120]

[0121]

[0122] Where max(Z) and max_id(Z) represent the maximum component and index of Z respectively, and N is the total number of categories. For binary classification b = [pmax ,p not_max Knowledge distillation is performed using KL (Knowledge Link-Based Language) as a method for maximum response knowledge distillation.

[0123] L max_cls =KL(b T ||b S (5)

[0124] use For other categories representing non-maximum responses, knowledge distillation is also performed using KL (Knowledge Distillation) to form the knowledge distillation method for non-maximum response classes as follows:

[0125]

[0126] Distillation loss in high-value distillation zones is defined as follows:

[0127]

[0128] Low-value classification distillation regions are more likely to be background in the image to be detected. Therefore, in order to better preserve this background-biased knowledge, this application uses the L1 norm of the difference between the teacher detector and the student detector's response to the original category in the new data as the target for fast optimization.

[0129]

[0130] In the knowledge distillation of the location branch, in order to fully explore the knowledge of different value regions, the embodiments of this application adopt different temperatures for location knowledge distillation for high and low value regions.

[0131] Using higher distillation temperatures in high-value location distillation areas will allow different response values ​​in the location distribution to be considered, as follows:

[0132]

[0133] Using a slightly lower distillation temperature in the low-value localization distillation region allows the student detector to focus more on the higher component values ​​in the response distribution of the low-value region, thereby avoiding noise interference from the low-value response values.

[0134]

[0135] Among them, B T and B S T1 and T2 are the localization responses of the teacher detector and student detector, respectively, and T1 and T2 are the temperatures of knowledge distillation. During the specific selection process, T2 must be kept < T1.

[0136] Finally, the refined knowledge distillation losses proposed in the embodiments of this application are summarized as follows:

[0137]

[0138] In summary, the incremental target detection method based on fine response distillation provided in this application can improve the detector's ability to retain knowledge of existing categories when learning new categories, and enhance the continuous learning ability of conventional target detectors in real dynamic scenarios. This incremental learning approach can expand the adaptability of conventional detectors to dynamically growing categories to be detected, broaden the application scenarios of target detectors, improve the efficiency of incremental learning, and can be further applied to incremental target detection in specific business scenarios.

[0139] See Figure 6 This application embodiment can also provide an incremental target detection device based on fine response distillation, such as... Figure 6 As shown, the device may include:

[0140] The teacher detector acquisition unit 601 is used to acquire the target detector of the knowledge to be distilled as the teacher detector; the teacher detector is a detector that has been learned on the original category data;

[0141] The student detector acquisition unit 602 is used to copy the network structure of the teacher detector and expand the number of output channels of its classification branch to form a student detector that needs to learn new categories while maintaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned.

[0142] The candidate distillation region determination unit 603 is used to obtain training data by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification responses and localization responses respectively based on the training data; and to perform threshold segmentation on the classification score with the highest confidence in the classification response of the teacher detector, which is used as a common candidate distillation region for transferring classification knowledge and localization knowledge from the teacher detector.

[0143] The refined distillation region determination unit 604 is used to divide candidate classification distillation regions into high-value classification distillation regions and low-value classification distillation regions by using the classification score with the highest confidence in the classification response of the teacher detector as the distillation value quality index of the classification branch and by modeling with a bimodal Gaussian mixture distribution; and to divide candidate positioning distillation regions into high-value positioning distillation regions and low-value positioning distillation regions by using the joint representation of the distillation value index of the classification branch of the teacher detector and the information entropy of the positioning branch response distribution as the distillation value quality index of the positioning branch and by modeling with a bimodal Gaussian mixture distribution.

[0144] The classification branch knowledge distillation unit 605 is used to decouple the classification response of the teacher detector into a maximum class response and a non-maximum class response in the high-value classification distillation region, and to obtain knowledge from the high-value classification response of the teacher detector by using the maximum response class knowledge distillation method and the non-maximum response class knowledge distillation method respectively for the student detector; in the low-value classification distillation region, knowledge is transferred by using the L1 norm knowledge distillation method for the response difference between the student detector and the teacher detector.

[0145] The location branch knowledge distillation unit 606 is used to obtain location knowledge of the high-value area by distilling the responses of the student detector and the teacher detector at a first distillation temperature in the high-value location distillation area; and to obtain location knowledge of the low-value area by distilling the responses of the student detector and the teacher detector at a second distillation temperature in the low-value location distillation area; wherein the first distillation temperature is greater than the second distillation temperature.

[0146] The student detector training unit 607 is used to iteratively train the student detector using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value localization distillation region distillation method, and the low-value localization distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories. Embodiments of this application can also provide an incremental target detection device based on fine response distillation, the device including a processor and a memory.

[0147] The memory is used to store program code and transmit the program code to the processor;

[0148] The processor is used to execute the steps of the above-described incremental target detection method based on fine response distillation according to the instructions in the program code.

[0149] like Figure 7 As shown in the figure, an incremental target detection device based on fine response distillation provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0150] In the embodiments of this application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0151] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the incremental target detection method based on fine response distillation.

[0152] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0153] The target detector that acquires knowledge about distillation is used as the teacher detector; the teacher detector is a detector that has been learned on the original category data;

[0154] The network structure of the teacher detector is copied, and the number of output channels of its classification branch is expanded to form a student detector that needs to learn new categories while retaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned.

[0155] Training data is obtained by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification and localization responses based on the training data, respectively; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and segmented as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector;

[0156] The classification score with the highest confidence in the teacher detector's classification response is used as the distillation value quality index of the classification branch. Candidate classification distillation regions are divided into high-value and low-value classification distillation regions through bimodal Gaussian mixture distribution modeling. The joint representation of the distillation value index of the teacher detector's classification branch and the information entropy of the localization branch response distribution is used as the distillation value quality index of the localization branch. Candidate localization distillation regions are divided into high-value and low-value localization distillation regions through bimodal Gaussian mixture distribution modeling.

[0157] In the high-value classification distillation region, the classification response of the teacher detector is decoupled into a maximum class response and a non-maximum class response. The student detector is then subjected to a maximum response class knowledge distillation method and a non-maximum response class knowledge distillation method to obtain knowledge from the high-value classification response of the teacher detector. In the low-value classification distillation region, the difference in response between the student detector and the teacher detector is transferred using an L1 norm knowledge distillation method.

[0158] In the high-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a first distillation temperature to obtain location knowledge for the high-value region; in the low-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a second distillation temperature to obtain location knowledge for the low-value region; the first distillation temperature is greater than the second distillation temperature.

[0159] The student detector is iteratively trained using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

[0160] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0161] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0162] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0163] Of course, it should be noted that, Figure 7 The structure shown does not constitute a limitation on the incremental target detection device based on fine response distillation in the embodiments of this application. In practical applications, the incremental target detection device based on fine response distillation may include devices with higher resolution than... Figure 7 More or fewer components as shown, or combinations of certain components.

[0164] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described incremental target detection method based on fine response distillation.

[0165] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0167] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0168] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An incremental target detection method based on fine response distillation, characterized in that, include: The target detector that acquires knowledge about distillation is used as the teacher detector; the teacher detector is a detector that has been learned on the original category data; The network structure of the teacher detector is copied, and the number of output channels of its classification branch is expanded to form a student detector that needs to learn new categories while retaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned. Training data is obtained by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification and localization responses based on the training data, respectively; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and segmented as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector; The classification score with the highest confidence in the teacher detector's classification response is used as the distillation value quality index of the classification branch. Candidate classification distillation regions are divided into high-value and low-value classification distillation regions through bimodal Gaussian mixture distribution modeling. The joint representation of the distillation value index of the teacher detector's classification branch and the information entropy of the localization branch response distribution is used as the distillation value quality index of the localization branch. Candidate localization distillation regions are divided into high-value and low-value localization distillation regions through bimodal Gaussian mixture distribution modeling. In the high-value classification distillation region, the classification response of the teacher detector is decoupled into a maximum class response and a non-maximum class response. The student detector is then subjected to a maximum response class knowledge distillation method and a non-maximum response class knowledge distillation method to obtain knowledge from the high-value classification response of the teacher detector. In the low-value classification distillation region, the difference in response between the student detector and the teacher detector is transferred using an L1 norm knowledge distillation method. In the high-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a first distillation temperature to obtain location knowledge for the high-value region; in the low-value location distillation region, the response of the student detector and the teacher detector is distilled using location knowledge at a second distillation temperature to obtain location knowledge for the low-value region; the first distillation temperature is greater than the second distillation temperature. The student detector is iteratively trained using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

2. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, The distillation regions are finely divided from the response of the teacher detector, including high-value classification distillation regions, low-value classification distillation regions, high-value localization distillation regions, and low-value localization distillation regions; including: The threshold segmentation of the classification confidence of the teacher detector is used as a coarse selection scheme for valuable distillation regions, and candidate distillable regions corresponding to the classification response and localization response of the teacher detector are selected. High-value distillation regions and low-value distillation regions are obtained by extracting from the candidate distillable regions.

3. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, The distillation value quality index Q of the classification branch cls It can be expressed by the following formula: In the formula: C T Let σ be the classification response of the teacher detector, and σ be the Sigmoid activation function.

4. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, The distillation value quality index Q of the positioning branch loc It can be expressed by the following formula: In the formula: E(p) is the general distribution information entropy, M is the category length corresponding to each distribution, and G is the predicted localization response distribution on each feature node.

5. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, Distillation value quality index for the aforementioned classification branches Threshold segmentation is performed to obtain the same candidate distillation regions in both the classification and localization responses; Based on the candidate regions and A dual-modal Gaussian mixture distribution is used for modeling, and the optimal boundary value between the two Gaussian distributions after modeling is used as the dynamic threshold for refining the candidate region segmentation. Regions with quality values ​​greater than the dynamic threshold are designated as high-value distillation regions, while regions with quality values ​​lower than the dynamic threshold are designated as low-value distillation regions.

6. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, The maximum category response p in the maximum response knowledge distillation and non-maximum response knowledge distillation max Non-maximum category response p not_max They are respectively expressed by the following formulas: In the formula: max(Z) and max_id(Z) represent obtaining the maximum component and the index of the maximum component of Z respectively, and N is the total number of categories. This represents the response output of the high-value region classification branch.

7. The incremental target detection method based on fine response distillation according to claim 1, characterized in that, The overall loss function L of knowledge distillation distill It can be expressed by the following formula: In the formula: The distillation loss function represents the distillation loss function for high-value classification distillation regions. The distillation loss function represents the distillation loss function for low-value classification distillation regions. This represents the distillation loss function for high-value positioning distillation regions. This represents the distillation loss function for low-value distillation regions.

8. An incremental target detection device based on fine response distillation, characterized in that, include: The teacher detector acquisition unit is used to acquire the target detector of the knowledge to be distilled as the teacher detector. The teacher detector is a detector that has been learned on the original category data; The student detector acquisition unit is used to copy the network structure of the teacher detector and expand the number of output channels of its classification branch to form a student detector that needs to learn new categories while retaining the knowledge of the original categories. The localization branch of the student detector is structurally the same as that of the teacher detector, and the classification branch of the student detector can simultaneously output classification prediction results containing the original categories and the new categories to be incrementally learned. The candidate distillation region determination unit is used to obtain training data by sampling from the new target category data, so that the teacher detector and the student detector can generate corresponding classification and localization responses respectively based on the training data; the classification score with the highest confidence in the classification response of the teacher detector is thresholded and used as a common candidate distillation region for the transfer of classification knowledge and localization knowledge from the teacher detector; The refined distillation region determination unit is used to divide candidate classification distillation regions into high-value and low-value classification distillation regions by using the classification score with the highest confidence in the teacher detector classification response as the distillation value quality index of the classification branch and by modeling with a bimodal Gaussian mixture distribution; and by using the joint representation of the distillation value index of the teacher detector classification branch and the information entropy of the localization branch response distribution as the distillation value quality index of the localization branch and by modeling with a bimodal Gaussian mixture distribution, candidate localization distillation regions are divided into high-value localization distillation regions and low-value localization distillation regions. The classification branch knowledge distillation unit is used to decouple the classification response of the teacher detector into a maximum class response and a non-maximum class response in the high-value classification distillation region, and to extract knowledge from the high-value classification response of the teacher detector by using the maximum response class knowledge distillation method and the non-maximum response class knowledge distillation method respectively for the student detector; in the low-value classification distillation region, knowledge is transferred by using the L1 norm knowledge distillation method for the response difference between the student detector and the teacher detector. A localization branch knowledge distillation unit is used to obtain localization knowledge of the high-value region by using localization knowledge distillation at a first distillation temperature on the responses of the student detector and the teacher detector in the high-value localization distillation region; and to obtain localization knowledge of the low-value region by using localization knowledge distillation at a second distillation temperature on the responses of the student detector and the teacher detector in the low-value localization distillation region; wherein the first distillation temperature is greater than the second distillation temperature. The student detector training unit is used to iteratively train the student detector using the high-value classification distillation region distillation method, the low-value classification distillation region distillation method, the high-value location distillation region distillation method, and the low-value location distillation region distillation method, so that the student detector retains the detection capability of the teacher detector for the original target categories while learning to detect new target categories.

9. An incremental target detection device based on fine response distillation, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the incremental target detection method based on fine response distillation as described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the incremental target detection method based on fine response distillation as described in any one of claims 1-7.

Images