Incremental target detection method based on confidence degree driving

Abstract

The invention provides an incremental target detection method based on confidence driving, and the method comprises the steps: generating a region-level pseudo label for a new image through an old model, and enabling the region-level pseudo label to be used for marking the region of an old-class target object in the new image; combining the real label of the new image with the generated pseudo label to form an expanded detection target category label for training a new model; generating a confidence coefficient value of a pixel level for the new image through the old model, combining the confidence coefficient value with the expanded label, and correcting a background region in the new model; training the new model, maintaining the detection of the old category through the region-level pseudo label and the pixel-level confidence value distillation, and learning the new category at the same time. The method has the advantages of being low in calculation cost and easy to integrate, and is suitable for large-scale incremental target detection scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a confidence-driven incremental target detection method. Background Art

[0002] Visual object detection is a fundamental task in computer vision. It plays a vital role in many fields, such as autonomous driving, multimedia information retrieval, and scene understanding. Thanks to the progress of deep learning, high detection performance can be achieved on both one-stage and two-stage detectors. For all these detectors, they are trained on a set of labeled samples with a fixed number of object categories. However, generally, new training samples from known or unknown categories are gradually added as the application scenarios gradually change. In order to adapt the trained model to the new scenario, there are two possible options, retraining the model from scratch or gradually updating the model. However, retraining the model from scratch requires a lot of computing resources, and the old model samples are difficult to obtain due to high maintenance costs, privacy issues, and intellectual property issues. Therefore, a way to gradually update the trained model should be considered.

[0003] Assuming that the newly added samples are from known categories, the model update can be done by simple fine-tuning. However, when the new samples come from new target categories, the problem becomes challenging. Simply fine-tuning the trained model will lead to a deterioration in the model's ability to recognize targets of old categories. This problem is widely known as the "catastrophic forgetting" problem. In fact, since most neural network designs assume that data is provided once, this problem is widespread in various deep neural networks. Suppose there is a deep learning task that is divided into multiple stages, and the network model is trained at each stage. The EWC (Elastic Weight Conservation) method selectively maintains the network weights that are important to the trained model in the previous stage, while allowing other parts to adapt to the new task. In essence, this is an incremental learning method based on regularization. However, experiments show that EWC does not perform as well as those based on knowledge distillation on incremental object detection.

[0004] In recent studies, incremental object detection has been solved through a variety of knowledge distillation strategies, most of which are based on two-stage detectors. However, due to the background offset problem, the old model and the new model still conflict with each other, although the two have been aligned at different levels through different knowledge distillation losses. Therefore, it is difficult for the new model to strike a balance between maintaining the knowledge of the old model and adapting to new target categories. In addition, since distillation on candidate boxes is impossible, this distillation strategy is not feasible on a single-stage detector. In addition, unlike a two-stage detector, all categories (whether old or new) of a single-stage detector share a localization head. When distillation is performed directly on the network output, conflicts between the new and old models become inevitable. In view of this, it is urgent to study an efficient incremental object detection method for a single-stage detector. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a confidence-driven incremental target detection method, which is suitable for target detection scenarios where new categories are gradually introduced for training, effectively avoiding background offset problems, while reducing computational costs and being easy to integrate.

[0006] The present invention is implemented as follows: a confidence-driven incremental target detection method, the method comprising: S1. Use the old model to generate region-level pseudo labels for the new image, which are used to mark the regions of the old category target objects in the new image. S2, combining the real label of the new image with the generated pseudo label to form an expanded detection target category annotation for training the new model; S3, generate pixel-level confidence values ​​for the new image through the old model, merge them with the expanded labels, and correct the background area in the new model; S4. Train the new model by distilling region-level pseudo labels and pixel-level confidence values, maintaining the detection of old categories while learning new categories.

[0007] Furthermore, the step S1 specifically includes: Step S11: Input the new image into the trained old model In the above example, we generate the predicted , , ,in represents the prediction box, Indicates confidence, Indicates category; Step S12: Use the non-maximum suppression method NMS to filter the pseudo labels and obtain the filtered prediction box With category , the filtered prediction boxes and categories are retained as pseudo labels for new images, and unreliable region predictions are filtered out by using non-maximum suppression.

[0008] Furthermore, the NMS threshold and confidence threshold of the non-maximum suppression method NMS are set to 0.6 and 0.1 respectively.

[0009] Furthermore, the step S2 is specifically as follows: Step 21: merge the pseudo label of the new image with the real label to form an expanded detection target category annotation, that is, an enhanced annotation for new model training, which is expressed as:

[0010]

[0011] in, and Represent the bounding box and category in the real label of the new image respectively. and Represents the bounding box and category after enhanced annotation; Step 22: Change the position loss function and the classification loss function. The position loss function is updated as follows:

[0012] The classification loss function is updated as:

[0013] in, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, , , and c represent the hyperparameters in the loss function, represents the intersection-over-union ratio of the predicted area and the true area, and n represents the number of samples. Specifically, represents the Euclidean distance between the center points of the predicted bounding box and the true bounding box, c represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box, α represents the weight hyperparameter, and v represents the consistency parameter of the aspect ratio of the predicted bounding box and the true bounding box.

[0014] Furthermore, the step S3 is specifically as follows: By calculation and The confidence score is generated by the intersection and union ratio between them, and the confidence loss formula is defined as:

[0015]

[0016]

[0017]

[0018] in, Represents the confidence of the new model for the new image output, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, n represents the number of samples, Represents the confidence after enhanced annotation, represents the indicator function, Indicates the confidence of the old model output; Afterwards, the network loss function The new model is trained using the corrected network loss function, which is expressed as: .

[0019] Furthermore, the new model is a single-stage detector and performs the object detection task by sharing the localization head.

[0020] Furthermore, the method also includes the step of gradually introducing different categories in multiple stages of incremental target detection, wherein the pseudo labels generated by the old model are used for all stages.

[0021] The present invention has the following advantages: by combining the pseudo-labels at the region level and the confidence values ​​at the pixel level for distillation, the present invention can generate pseudo-labels and combine them with the real labels of the new category without replaying the old category samples, effectively solving the catastrophic forgetting problem in incremental target detection, ensuring that the new model can retain the ability to detect the target objects of the old category, and correcting the judgment of the new model on the background area by correcting the confidence of the background area, preventing the target objects of the old category from being misjudged as the background in the new image, and avoiding the background offset problem. While maintaining high detection accuracy, the present invention has a low computational cost and is suitable for large-scale incremental learning scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0023] Figure 1 The present invention is a flowchart of an execution of a confidence-driven incremental target detection method.

[0024] Figure 2 This is a framework diagram of a confidence-driven incremental target detection method of the present invention. DETAILED DESCRIPTION

[0025] The technical solution in the embodiment of the present application has the following general idea: In view of the problems existing in the prior art, the present invention uses distillation strategies at the regional level and the pixel level to ensure that the new model can learn new categories and avoid the loss of detection performance of the old categories. Similar to most incremental target detection methods, the update of the new model is based on knowledge distillation. Distillation at the regional level is achieved through the pseudo-labels of the old targets. In order to solve the problem of background offset when the pseudo-labels are not available, another distillation is achieved through the confidence score map. Specifically, when the method distills knowledge at the regional level, the old model is used to predict the target objects of the old category in the new image, generate pseudo-labels, use non-maximum suppression to filter unreliable predictions, and combine the pseudo-labels with the real labels of the new category to form enhanced annotations; when distilling at the pixel level, the confidence score generated by the old model is used to correct the background area judgment of the new model to avoid the background offset problem; the incremental target detection framework proposed by the present invention is suitable for single-stage detectors and has a wide range of applicability; compared with the existing incremental target detection methods, the present invention has the advantages of low computational cost and easy integration, and is suitable for large-scale incremental target detection scenarios.

[0026] In order to better understand the above technical solution, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] See also Figure 1 and 2 As shown, the present invention provides a confidence-driven incremental target detection method, the method comprising: S1. Use the old model to generate region-level pseudo labels for the new image, which are used to mark the regions of the old category target objects in the new image. S2, combining the real label of the new image with the generated pseudo label to form an expanded detection target category annotation for training the new model; S3, generate pixel-level confidence values ​​for the new image through the old model, merge them with the expanded labels, and correct the background area in the new model; S4. Train the new model by distilling region-level pseudo labels and pixel-level confidence values, maintaining the detection of old categories while learning new categories.

[0028] Preferably, the step S1 specifically includes: Step S11: Input the new image into the trained old model In the above example, we generate the predicted , , ,in represents the prediction box, Indicates confidence, Indicates category; Step S12: Use the non-maximum suppression method NMS to filter the pseudo labels and obtain the filtered prediction box With category , the filtered prediction boxes and categories are retained as pseudo labels for new images, and unreliable region predictions are filtered out by using non-maximum suppression.

[0029] Preferably, the NMS threshold and confidence threshold of the non-maximum suppression method NMS are set to 0.6 and 0.1 respectively.

[0030] Preferably, the step S2 is specifically as follows: Step 21: merge the pseudo label of the new image with the real label to form an expanded detection target category annotation, that is, an enhanced annotation for new model training, which is expressed as:

[0031]

[0032] in, and Represent the bounding box and category in the real label of the new image respectively. and Represents the bounding box and category after enhanced annotation; Step 22: Change the position loss function and the classification loss function. The position loss function is updated as follows:

[0033] The classification loss function is updated as:

[0034] in, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, , , and c represent the hyperparameters in the loss function, represents the intersection-over-union ratio of the predicted area and the true area, and n represents the number of samples. Specifically, represents the Euclidean distance between the center points of the predicted bounding box and the true bounding box, c represents the diagonal length of the minimum closure (i.e., the minimum enclosing rectangle) containing the predicted bounding box and the true bounding box, and α represents a weight hyperparameter used to balance the center point distance As with other terms in the CIoU loss, it is usually a dynamic value that depends on the value of IoU to ensure that when IoU is high, the center point distance term has a greater impact on the loss. Indicates the consistency of the aspect ratio of the predicted bounding box and the true bounding box. This term penalizes the situation where the aspect ratio of the predicted bounding box and the true bounding box are inconsistent.

[0035] Preferably, the step S3 is specifically as follows: By calculation and The confidence score is generated by the intersection and union ratio between them, and the confidence loss formula is defined as:

[0036]

[0037]

[0038]

[0039] Where n represents the number of samples, Represents the confidence of the new model for the new image output, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, Represents the confidence after enhanced annotation, represents the indicator function, Indicates the confidence of the old model output; Afterwards, the network loss function The new model is trained using the corrected network loss function, which is expressed as: .

[0040] Preferably, the new model is a single-stage detector and performs the object detection task by sharing the localization head.

[0041] Preferably, the method further comprises the step of gradually introducing different categories in multiple stages of incremental target detection, wherein the pseudo labels generated by the old model are used for all stages.

[0042] In order to better understand the above technical solution, the definitions involved in the incremental target detection process are described in detail below.

[0043] First, the definition of general incremental object detection is as follows: Set the target set to C, and these categories are gradually introduced into the detection model from set C. Subset Introduced into the detection model at stage t. For any ,if ,but .set up Indicates that it contains A collection of images with labeled target objects of different categories in . ∈ can contain objects from both new and old categories. However, regarding the old category objects, i.e. The annotation will not be provided. Also, replaying the image is not allowed. .

[0044] The goal of incremental object detection is to build a series of detection models , assuming that the current training stage is t, the student model (i.e. new model) based on image set Perform training, given a trained model Because the teacher model (i.e., the old model) is expected to be able to Therefore, only the student model is trained Teacher model is needed When t=2, incremental detection is usually called single-stage incremental detection; when t>2, it is called multi-stage incremental detection.

[0045] For ease of understanding, a typical single-stage detector YOLOv7 is used for analysis. The key components of YOLOv7 are: YOLOv7 logits bounding box given model , confidence and categories , and their corresponding true values ​​are , and The training goal of YOLOv7 is to minimize the bounding box regression loss defined on the model logits. , target confidence loss and positive sample classification loss . Network loss Defined as:

[0046] Bounding Box Regression Loss , target confidence loss and positive sample classification loss The formula is as follows:

[0047] In the above loss function, is the loss between the predicted box and the true bounding box calculated after the positive sample is matched, and is a hyperparameter in the loss function. Similarly, the confidence loss and classification loss It is calculated by cross entropy.

[0048] 1 When performing the zone level distillation of the present invention: 1.1 Input image to generate pseudo labels Assuming the old model Works well with all observed categories . Appears in the image ∈ The old category target objects on can be regarded as Towards a new model A convenient medium for transferring knowledge. Basically, the old model is used as an automatic annotator to annotate images Old categories on Generate labels. By ∈ Input to model , we get:

[0049] At this point, the generated prediction value All potential areas where the old category target objects may appear are specified. In order to filter out unreliable predictions, we further use the non-maximum suppression (NMS) method Get the filtered and ,In the NMS operation, the NMS threshold and the confidence threshold are set to 0.6 and 0.1, respectively.

[0050] 1.2 Combining Pseudo-Labels with True Labels to Generate Enhanced Annotations The predicted region is taken as an image The pseudo labels on the image are retained. The real labels provided are merged to form a dataset After all images have been processed in this way, the enhanced annotations can be expressed as:

[0051] Since these enhanced annotations will be used to train the model The position loss and classification loss functions are modified to and Since the labels of the old categories are involved in the training, the existence of the old category objects no longer interferes with the training of the new model. Instead, the new model will be compatible with the old category objects.

[0052] 2 When performing the regional level distillation of the present invention: The region-level distillation mentioned above makes full use of the target objects of the old categories that appear in the new sample images, basically solving the background shift problem. However, these target objects do not necessarily cover all observed categories. In extreme cases, there may not be any target objects of the old categories in the new sample images. Therefore, the new model The detection ability for these missing categories will inevitably be lost. This situation will gradually worsen with the increase of training stages.

[0053] For two-stage detectors, it is easy to perform direct distillation on the network output. However, for single-stage detectors such as YOLO, this is basically infeasible because the new and old categories share the same localization head. If we choose to distill the feature map, we will face a similar situation as background shift. arrive The knowledge distillation method is applicable to single-stage detectors.

[0054] 2.1 Generating Confidence Scores From the old model In the image The confidence scores generated from t largely make up for the shortcomings of pseudo-labels, especially when pseudo-labels are not available. More importantly, these scores are available in both one-stage and two-stage detectors. Therefore, in addition to pseudo-labels, we also obtain Towards Distill the confidence scores. However, if you inherit the old model directly The entire confidence score map of will be superimposed with the confidence scores generated by the enhanced annotation. To solve this problem, by calculating and , because the enhanced annotations are already available. In order to distill the confidence from Deliver to , the confidence loss formula is modified as follows:

[0055] Where I is the indicator function. Confidence score The true value is calculated by the bounding box With the true value The intersection and union between them is obtained. When , it indicates that there is a target object at the current position. , it means that the pixel belongs to the background. In this case, The generated confidence scores will be directly assigned to In this way, The knowledge in We first mark the enhanced annotation region as the target region ( ), and then the background area ( ) replaced with the old model The confidence score generated is used to correct the background area.

[0056] 2.2 Correcting the network loss function Corrected confidence loss It integrates the distillation at the region level and pixel level. When substituting into YOLOv7, its network loss function is corrected to:

[0057] Through the above-mentioned region-level pseudo-label and pixel-level confidence value distillation, the detection of the old categories is maintained during the training of the new model, while learning new categories. The method of the present invention is simple and efficient, and is suitable for single-stage detector confidence distillation. Teacher Model The knowledge is distilled at two levels. At the regional level, Generate pseudo labels and Merge the real labels to generate enhanced real labels At the pixel level, the confidence score Requirements and confidence scores To combine with region-level distillation, the regions corresponding to the enhanced true labels are retained. Since both region-level and pixel-level distillation are integrated into the confidence score, our method is also called confidence score distillation (CSD).

[0058] The performance of the method (CSD) of the present invention is tested below: In the first experiment, experiments were conducted on the currently popular target detection test dataset Pascal-VOC 2007. The experimental results are shown in Table 1 below: Table 1

[0059] In this experiment, we verified the effectiveness of our method on a single-stage learning task. The method of the present invention, CSD, i.e., confidence score distillation, was integrated into the single-stage detector YOLOv7 for testing. Similarly, other methods OnlineOD, EWC, and KD were also integrated into YOLOv7, respectively. Table 1 shows the performance results on VOC 2007. Three single-stage incremental learning settings were tested, namely 10+10, 15+5, and 19+1, respectively. The first 19, 15, and 10 categories in the training set were regarded as old categories, corresponding to the three evaluation settings, respectively, and the remaining 1, 5, or 10 categories were regarded as new categories. As can be seen from Table 1, the performance of CSD on YOLOv7 is significantly better than other methods.

[0060] In the second experiment, we studied the effectiveness of the CSD method of the present invention in the multi-stage incremental object detection task when it is integrated into the YOLOv7 model. As shown in Table 2 below, the object categories in the Pascal VOC 2007 and COCO 2017 datasets are evenly divided into four groups: A, B, C and D. The experiment divides the incremental training into four stages, namely: starting from stage A, gradually adding new categories in stage B, and gradually adding new object categories in stage C and stage D for training. As can be seen from Table 2, on the Pascal VOC 2007 and COCO 2017 datasets, the best performance comes from the CSD method based on YOLOv7. On the one hand, this is largely due to the excellent performance of YOLOv7, which provides a good foundation for the incremental detector; on the other hand, it also shows that the distillation strategy we proposed successfully transfers old knowledge to the new model.

[0061] Table 2

[0062] Through experimental verification on Pascal VOC 2007 and COCO 2017 datasets, the method of the present invention can significantly improve the detection accuracy of old categories compared with existing methods while maintaining the detection performance of new categories. In addition, the present invention has strong adaptability in multi-stage incremental learning and has good feasibility and effectiveness for target detection scenarios in practical applications.

[0063] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A confidence-driven incremental target detection method, characterized in that: The method comprises: S1. Use the old model to generate region-level pseudo labels for the new image, which are used to mark the regions of the old category target objects in the new image. S2, combining the real label of the new image with the generated pseudo label to form an expanded detection target category annotation for training the new model; S3, generate pixel-level confidence values ​​for the new image through the old model, merge them with the expanded labels, and correct the background area in the new model; S4. Train the new model by distilling region-level pseudo labels and pixel-level confidence values, maintaining the detection of old categories while learning new categories.

2. The confidence-driven incremental target detection method according to claim 1, characterized in that: The step S1 specifically includes: Step S11: Input the new image into the trained old model In the above example, we generate the predicted , , ,in represents the prediction box, Indicates confidence, Indicates category; Step S12: Use the non-maximum suppression method NMS to filter the pseudo labels and obtain the filtered prediction box With category , the filtered prediction boxes and categories are retained as pseudo labels for new images, and unreliable region predictions are filtered out by using non-maximum suppression.

3. The confidence-driven incremental target detection method according to claim 2, characterized in that: The NMS threshold and confidence threshold of the non-maximum suppression method NMS are set to 0.6 and 0.1 respectively.

4. The confidence-driven incremental target detection method according to claim 1, characterized in that: The step S2 is specifically as follows: Step 21: merge the pseudo label of the new image with the real label to form an expanded detection target category annotation, that is, an enhanced annotation for new model training, which is expressed as: in, and Represent the bounding box and category in the real label of the new image respectively. and Represents the bounding box and category after enhanced annotation; Step 22: Change the position loss function and the classification loss function. The position loss function is updated as follows: The classification loss function is updated as: in, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, It represents the intersection-over-combination ratio of the predicted area and the true area. represents the Euclidean distance between the center points of the predicted bounding box and the true bounding box, c represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box, α represents the weight hyperparameter, v represents the consistency parameter of the aspect ratio of the predicted bounding box and the true bounding box, and n represents the number of samples.

5. The confidence-driven incremental target detection method according to claim 1, characterized in that: The step S3 is specifically as follows: By calculation and The confidence score is generated by the intersection and union ratio between them, and the confidence loss formula is defined as: Where n represents the number of samples, Represents the confidence of the new model for the new image output, Represents the prediction box output by the new model for the new image, Represents the category of the new model output for the new image, Represents the confidence after enhanced annotation, represents the indicator function, Indicates the confidence of the old model output; Afterwards, the network loss function The new model is trained using the corrected network loss function, which is expressed as: 。 6. The confidence-driven incremental target detection method according to claim 1, characterized in that: The new model is a single-stage detector and performs the object detection task by sharing the localization head.

7. The confidence-driven incremental target detection method according to claim 1, characterized in that: The method also includes the step of gradually introducing different categories in multiple stages of incremental object detection, where pseudo labels generated by the old model are used for all stages.

Images