An open world object detection method based on adaptive semantic degradation learning

Abstract

The application provides an open world target detection method based on adaptive semantic degradation learning, belongs to the technical field of computer vision, is suitable for scenarios such as automatic driving, and comprises the following steps: using known class data of an automatic driving scene with labeled information to guide an open target detection model in a current stage to detect known class targets of the automatic driving; guiding the open target detection model to obtain shared features and information between different clustering clusters; using the open target detection model to distinguish known classes and unknown classes with shared generalization features; setting to execute update of a training data set, guiding the open target detection model to perform incremental learning of new classes, completing training of the open target detection model, and detecting targets in an open world of an automatic driving scene. The application can be used in an automatic driving scene, is also suitable for open world target detection tasks such as defect detection and target tracking, and has high practical application value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to an open-world object detection method based on adaptive semantic degradation learning. Background Technology

[0002] Open-world object detection aims to detect objects in images of open-world scenes. Unlike object detection in conventional closed-set scenes, open-world object detection simulates the ever-growing knowledge environment of a real dynamic world. It requires algorithms to detect not only known classes of objects in the image but also unknown classes not included in the training set, and to incrementally learn the identified unknown classes based on dataset updates. The open-world setting has attracted widespread attention in recent years. Existing techniques first proposed an experimental setting for open-world object detection, which includes multiple incremental tasks. In each task, the detection model trained only on known classes should correctly detect known classes during the testing phase and identify unknown classes as "unknown." Then, human annotators can progressively assign labels to classes of interest, and the detection model in the next task should progressively learn these classes using the newly added labels.

[0003] Compared with traditional closed-set object detection tasks, open-world object detection tasks present new challenges: (1) Detection of unknown classes: detect unknown autonomous driving scene object instances and distinguish them from similar known autonomous driving scene object instances and the background. (2) Incremental learning: be able to incrementally learn the identified unknown classes and achieve a balance between learning the original known classes and the newly labeled known classes.

[0004] Existing technologies utilize an unknown class candidate region generation network with an automatic labeling strategy and incorporate an energy-based binary classifier to distinguish between unknown and known classes. These technologies predefine a semantic centroid for each class and, during incremental learning, push the target instance of the autonomous driving scene closer to its centroid to enhance the distinction between unknown and known classes. They also incorporate attention-driven pseudo-labeling, novelty classification, and object scoring methods into the DETR model for unknown class detection. Furthermore, these technologies supplement and refine the benchmarks and evaluation metrics for object detection tasks in open-world settings and employ a non-parametric candidate box guidance module and a class-specific exclusionary classifier to improve unknown class detection.

[0005] Although the above methods achieve the detection of unknown class objects through specific network structures, the detection performance of these methods still needs to be improved because in open-world object detection tasks, there is only a large amount of known class annotation information in the dataset, while the detection of unknown classes depends on the unknown class candidate box generation mechanism with high uncertainty. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides an open-world object detection method based on adaptive semantic degradation learning. This invention can be applied to object detection scenarios in open worlds such as autonomous driving, and has high practical application value.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: an open-world object detection method based on adaptive semantic degradation learning, comprising the following steps:

[0008] S1. Utilize known class data of autonomous driving scenarios with labeled information to guide the current open target detection model to detect known class targets in autonomous driving.

[0009] S2. Using semantic degradation learning methods, guide open object detection models to acquire shared features and information between different clusters;

[0010] S3. Based on the acquired shared features and information, use an open target detection model to distinguish between known and unknown categories of shared generalized features;

[0011] S4. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset and guide the open object detection model to perform incremental learning of new classes;

[0012] S5. Iterate through steps S1-S4 to complete the training of the open object detection model and use the trained open object detection model to detect objects in the open world of the autonomous driving scenario.

[0013] Further, step S2 includes the following steps:

[0014] S201. Based on the detection results of known target classes in the autonomous driving scenario, divide each known category into C clusters;

[0015] S202. Based on the C clusters, assign cluster-level labels to all known true labels Tt and high-confidence known category false alarms Pt for autonomous driving scene target instances. For true autonomous driving scene target instances with known categories, assign cluster labels corresponding to their true categories. For false autonomous driving scene target instances with known categories, assign cluster labels corresponding to their predicted categories.

[0016] S203. Based on the open object detection model, a semantic reduction learning method is used to construct a semantic space in the assigned cluster level to obtain shared features and information between different clusters. The classification regularization expression for semantic reduction learning is as follows:

[0017]

[0018] in, The label represents the target instance i in the autonomous driving scenario after semantic degradation. This represents the clustering label of the known true category label of target instance i in an autonomous driving scenario after semantic degradation. The clustering label represents the predicted false positive label of target instance i in the autonomous driving scenario after semantic downgrading. This represents the new label of the cluster to which target instance i in the autonomous driving scenario belongs after semantic degradation. This represents the true label of target instance i in the autonomous driving scenario. τ represents the predicted label of target instance i in the autonomous driving scenario, τ represents the set of real labeled target instances in the autonomous driving scenario, ρ represents the set of false positives predicted with high confidence, and i represents the index of the target instance in the autonomous driving scenario.

[0019] The classification regularization for semantic degradation can be expressed as:

[0020]

[0021] in, This represents classification regularization for semantic degradation learning. This represents the union of the set of known-category real-world autonomous driving scenario target instances and the set of predicted false positives.

[0022] Furthermore, step S3 specifically includes:

[0023] Based on the acquired shared features and information, the first-layer output of the decoder in the known category branch is fused with the output of each layer in the unknown category branch, and discriminative information is introduced to distinguish between known and unknown categories that share generalization features. The discriminative process is as follows:

[0024] E′ unknown，j =E unknown，j +α j ·E known,1

[0025] Among them, E' unknown,j E represents the fused embedding output of the unknown category branch in the j-th layer. unknown,j E represents the output feature of the j-th layer in the unknown class detection branch. unknown,ja represents the output feature of the first layer in the known class detection branch. j This represents the adaptive parameter of the j-th layer, used to adjust the degree of fusion between known categorical features and generalized features.

[0026] Furthermore, step S4 includes the following steps:

[0027] S401. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset;

[0028] S402. Retrain the open object detection model using the updated open-world object detection training set for autonomous driving to adapt to the open object detection task in the new stage.

[0029] S403. Adaptive semantic degradation learning is performed on the updated open-world target detection training set for autonomous driving to guide the open target detection model in the new stage to learn collaboratively between known and unknown classes.

[0030] S404. Using the updated open-world target detection training set for autonomous driving, distinguish between known and unknown classes in the open target detection model of the new stage of enhanced guidance for unknown classes based on discriminative guidance.

[0031] S405. Perform small-sample fine-tuning training on the open target detection model for the new stage of autonomous driving open world targets, guiding the model to learn to detect targets from all previous stages simultaneously.

[0032] S406. Based on the processing results of steps S401-S405, complete the new class incremental learning of the open target detection model.

[0033] Furthermore, step S405 includes the following steps:

[0034] S4051, Small sample set D for autonomous driving target detection in construction phase t ex Among them, the small sample set D for autonomous driving target detection ex N including all known categories in the first t stages ex One sample;

[0035] S4052. In stage t+1, after completing the incremental learning of autonomous driving target detection using the updated autonomous driving target detection training set, the autonomous driving target detection small sample set D is used. ex Fine-tuning and training of the open target detection model for the new stage.

[0036] Furthermore, step S5 includes the following steps:

[0037] S501. Iteratively execute steps S1-S4 to complete the training of the open target detection model;

[0038] S502, Acquire images of the autonomous driving scene to be detected;

[0039] S503. Based on the autonomous driving scene image to be detected, multi-scale image features are obtained using the pyramid extractor in the trained open object detection model.

[0040] S504. Encode multi-scale image features to obtain high-level semantic features;

[0041] S505. Input the high-level semantic features into the known category branch to obtain the known class results, and perform semantic downgrading on the known classes to obtain cluster-level classification;

[0042] S506. Perform target detection on known class results and cluster-level results, and delete target detection results that are greater than a preset threshold to obtain the detection of unknown classes, thus completing the detection of targets in the open world of autonomous driving scenarios.

[0043] The beneficial effects of this invention are:

[0044] This invention provides an open-world object detection method based on adaptive semantic degradation learning. Based on the principle that known and unknown classes share common semantic features and information, it extracts effective common semantic information from different clusters in the data to guide the learning of unknown knowledge. Furthermore, by adaptively aligning the feature space, it guides the open object detection model to accurately distinguish between known and unknown categories, effectively improving the detection performance of the open object detection model for unknown categories while retaining its strong detection capability for the original known categories. This invention can be applied to tasks such as autonomous driving. The design of this invention closely resembles real-world scenarios, has high practical application value, and has achieved excellent performance. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention. Figure 2 This is a visual example of an autonomous driving task. Detailed Implementation

[0046] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0047] Example 1

[0048] like Figure 1 As shown, this invention provides an open-world object detection method based on adaptive semantic degradation learning, the implementation of which is as follows:

[0049] S1. Utilize known class data of autonomous driving scenarios with labeled information to guide the current open target detection model to detect known class targets in autonomous driving.

[0050] S2. Using semantic degradation learning methods, guide open object detection models to acquire shared features and information between different clusters;

[0051] S3. Based on the acquired shared features and information, use an open target detection model to distinguish between known and unknown categories of shared generalized features;

[0052] S4. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset and guide the open object detection model to perform incremental learning of new classes;

[0053] S5. Iterate through steps S1-S4 to complete the training of the open object detection model, and use the trained open object detection model to detect objects in the open world of the autonomous driving scenario.

[0054] In this embodiment, step S2 includes the following steps:

[0055] S201. Based on the detection results of known target classes in the autonomous driving scenario, divide each known category into C clusters;

[0056] S202. Based on the C clusters, assign cluster-level labels to all known true labels Tt and high-confidence known category false alarms Pt for autonomous driving scene target instances. For true autonomous driving scene target instances with known categories, assign cluster labels corresponding to their true categories. For false autonomous driving scene target instances with known categories, assign cluster labels corresponding to their predicted categories.

[0057] S203. Based on the open object detection model, a semantic reduction learning method is used to construct a semantic space in the assigned cluster level to obtain shared features and information between different clusters. The classification regularization expression for semantic reduction learning is as follows:

[0058]

[0059] in, The label represents the target instance i in the autonomous driving scenario after semantic degradation. This represents the clustering label of the known true category label of target instance i in an autonomous driving scenario after semantic degradation. The clustering label represents the predicted false positive label of target instance i in the autonomous driving scenario after semantic downgrading. This represents the new label of the cluster to which target instance i in the autonomous driving scenario belongs after semantic degradation. This represents the true label of target instance i in the autonomous driving scenario. τ represents the predicted label of target instance i in the autonomous driving scenario, τ represents the set of real labeled target instances in the autonomous driving scenario, ρ represents the set of false positives predicted with high confidence, and i represents the index of the target instance in the autonomous driving scenario.

[0060] The classification regularization for semantic degradation can be expressed as:

[0061]

[0062] in, This represents classification regularization for semantic degradation learning. This represents the union of the set of known-category real-world autonomous driving scenario target instances and the set of predicted false positives.

[0063] In this embodiment, step S3 specifically includes:

[0064] Based on the acquired shared features and information, the first-layer output of the decoder in the known category branch is fused with the output of each layer in the unknown category branch, and discriminative information is introduced to distinguish between known and unknown categories that share generalization features. The discriminative process is as follows:

[0065] E′ unknown，j =E unknown，j +α j ·E known，1

[0066] Among them, E' unknown,j E represents the fused embedding output of the unknown category branch in the j-th layer. unknown,j E represents the output feature of the j-th layer in the unknown class detection branch. unknown,j a represents the output feature of the first layer in the known class detection branch. j This represents the adaptive parameter of the j-th layer, used to adjust the degree of fusion between known categorical features and generalized features.

[0067] In this embodiment, step S4 includes the following steps:

[0068] S401. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset;

[0069] S402. Retrain the open object detection model using the updated open-world object detection training set for autonomous driving to adapt to the open object detection task in the new stage.

[0070] S403. Adaptive semantic degradation learning is performed on the updated open-world target detection training set for autonomous driving to guide the open target detection model in the new stage to learn collaboratively between known and unknown classes.

[0071] S404. Using the updated open-world target detection training set for autonomous driving, distinguish between known and unknown classes in the open target detection model of the new stage of enhanced guidance for unknown classes based on discriminative guidance.

[0072] S405. Perform small-sample fine-tuning training on the open target detection model for the new stage of autonomous driving open world targets, guiding the model to learn to detect targets from all previous stages simultaneously.

[0073] S406. Based on the processing results of steps S401-S405, complete the new class incremental learning of the open target detection model.

[0074] In this embodiment, step S405 includes the following steps:

[0075] S4051, Small sample set D for autonomous driving target detection in construction phase t ex Among them, the small sample set D for autonomous driving target detection ex N including all known categories in the first t stages ex One sample;

[0076] S4052. In stage t+1, after completing the incremental learning of autonomous driving target detection using the updated autonomous driving target detection training set, the autonomous driving target detection small sample set D is used. ex Fine-tuning and training of the open target detection model for the new stage.

[0077] In this embodiment, step S5 includes the following steps:

[0078] S501. Iteratively execute steps S1-S4 to complete the training of the open target detection model;

[0079] S502, Acquire images of the autonomous driving scene to be detected;

[0080] S503. Based on the autonomous driving scene image to be detected, multi-scale image features are obtained using the pyramid extractor in the trained open object detection model.

[0081] S504. Encode multi-scale image features to obtain high-level semantic features;

[0082] S505. Input the high-level semantic features into the known category branch to obtain the known class results, and perform semantic downgrading on the known classes to obtain cluster-level classification;

[0083] S506. Perform target detection on known class results and cluster-level results, and delete target detection results that are greater than a preset threshold to obtain the detection of unknown classes, thus completing the detection of targets in the open world of autonomous driving scenarios.

[0084] In this embodiment, an autonomous driving image X to be detected is input into the trained target detection model. A pyramid feature extractor is used to obtain multi-scale image features, which are then encoded to obtain rich semantic features. N s M represents the length of the semantic features, and D represents the dimension of the transformer. Inputting M into the known class detection branch yields known class results, while inputting M into the unknown class detection branch yields cluster-level results for generalized objects. The latter contains a certain number of known and unknown targets. Subsequently, the IOU between the results detected by the unknown class branch and the targets detected by the known class branch is performed; categories exceeding a certain threshold are directly filtered out to obtain the detection results for the unknown classes.

[0085] In this embodiment, steps S1 to S4 are executed iteratively. In one embodiment of the invention, incremental learning is set to four stages. Each new stage needs to learn new known categories while retaining the detection capability for the original known categories, and can also detect target instances in unknown autonomous driving scenarios. The model uses the SGD optimizer, and the batch size is set to 8. For the hyperparameter, the number of clusters C is set to 3.

[0086] This invention conducts experiments on an open-world object detection dataset for autonomous driving to verify the practical effectiveness of the open-world object detection algorithm for autonomous driving provided in this embodiment. Specifically, image data from the OWOD dataset, which is adapted to autonomous driving scenarios, was selected, containing images of 80 different object categories.

[0087] To quantitatively and accurately measure the performance of the proposed algorithm, the mean accuracy (K-mAP) of known classes, the mean accuracy (U-mAP) of unknown classes, and the recall (U-Recall) of unknown classes in autonomous driving scenarios were used as metrics. The proposed algorithm was fairly compared with other methods, and the results from the first to the fourth stages are shown in Table 1. As can be seen from Table 1, compared with other methods, the proposed algorithm achieves superior performance in detecting both known and unknown classes in autonomous driving scenarios. Specifically, the mean accuracy of unknown classes in Stage 1 was improved by 50%, demonstrating greater practical value in autonomous driving tasks.

[0088] Table 1

[0089]

[0090]

[0091] In this embodiment, a visualization example of an autonomous driving task is as follows: Figure 2 As shown, the algorithm proposed in this invention can not only detect known classes such as cars and pedestrians in open road scenes, but also detect unlearned bridge deck barriers. Figure 2 Left), tire base ( Figure 2 (in the middle) and skateboards in the middle of the road ( Figure 2 (Right) and mark it as an unknown class to achieve collaborative recognition of known and unknown targets in open-world natural scene images, thereby avoiding unexpected situations that may occur in the autonomous driving process due to the appearance of unknown objects.

[0092] Inspired by cognitive development theory, this invention guides the model to capture more general semantic features through feature patterns learned through natural downgrading. This enables the discovery of unknown targets that share these features with different known categories, effectively improving the model's detection performance for unknown categories while retaining its strong detection capabilities for known categories. The invention's design closely resembles real-world scenarios, possessing high practical application value and achieving excellent performance.

Claims

1. An open-world object detection method based on adaptive semantic degradation learning, characterized in that, Includes the following steps: S1. Utilize known class data of autonomous driving scenarios with labeled information to guide the current open target detection model to detect known class targets in autonomous driving. S2. Using semantic degradation learning methods, guide open object detection models to acquire shared features and information between different clusters; Step S2 includes the following steps: S201. Based on the detection results of known target classes in the autonomous driving scenario, divide each known category into C clusters; S202. Based on the C clusters, assign cluster-level labels to all known true labels Tt and high-confidence known category false alarms Pt for autonomous driving scene target instances. For true autonomous driving scene target instances with known categories, assign cluster labels corresponding to their true categories. For false autonomous driving scene target instances with known categories, assign cluster labels corresponding to their predicted categories. S203. Based on the open object detection model, a semantic degradation learning method is used to construct a semantic space in the assigned cluster level to obtain shared features and information between different clusters. The classification regularization expression for semantic degradation learning is as follows: in, Represents target instances in an autonomous driving scenario after semantic degradation. The tag, Represents target instances in autonomous driving scenarios The clustered labels of the known true category labels after semantic degradation Represents target instances in autonomous driving scenarios The clustered labels after semantic downgrading of the predicted false positive labels. Represents target instances in autonomous driving scenarios The new label of the cluster after semantic downgrading Represents target instances in autonomous driving scenarios The true label, Represents target instances in autonomous driving scenarios Predicted labels, This represents the set of target instances in a real-world, labeled autonomous driving scenario. This represents the set of false positives in high-confidence predictions. An index representing a target instance in an autonomous driving scenario; The classification regularization for semantic degradation can be expressed as: in, This represents classification regularization for semantic degradation learning. This represents the union of the set of known-category real-world autonomous driving scenario target instances and the set of predicted false positives; S3. Based on the acquired shared features and information, use an open target detection model to distinguish between known and unknown categories of shared generalized features; S4. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset and guide the open object detection model to perform incremental learning of new classes; S5. Iterate through steps S1-S4 to complete the training of the open object detection model and use the trained open object detection model to detect objects in the open world of the autonomous driving scenario.

2. The open-world object detection method based on adaptive semantic degradation learning according to claim 1, characterized in that, Step S3 specifically involves: Based on the acquired shared features and information, the first-layer output of the decoder in the known category branch is fused with the output of each layer in the unknown category branch, and discriminative information is introduced to distinguish between known and unknown categories that share generalization features. The discriminative process is as follows: in, Indicates the first j The fusion embedding output of the unknown category branch of the layer, Indicating the unknown class detection branch, the first j The output features of the layer This represents the output features of the first layer in the known class detection branch. Indicates the first j The adaptive parameters of the layer are used to adjust the degree of fusion between known categorical features and generalized features.

3. The open-world object detection method based on adaptive semantic degradation learning according to claim 1, characterized in that, Step S4 includes the following steps: S401. Based on the open-world object detection task in the autonomous driving scenario, set up the update of the training dataset; S402. Retrain the open object detection model using the updated open-world object detection training set for autonomous driving to adapt to the open object detection task in the new stage. S403. Adaptive semantic degradation learning is performed on the updated open-world target detection training set for autonomous driving to guide the open target detection model in the new stage to perform collaborative learning of known and unknown classes. S404. Using the updated open-world target detection training set for autonomous driving, distinguish between known and unknown classes in the open target detection model of the new stage of enhanced guidance for unknown classes based on discriminative guidance. S405. Perform small-sample fine-tuning training on the open target detection model for the new stage of autonomous driving open world targets, guiding the model to learn to detect targets from all previous stages simultaneously. S406. Based on the processing results of steps S401-S405, complete the new class incremental learning of the open target detection model.

4. The open-world object detection method based on adaptive semantic degradation learning according to claim 3, characterized in that, Step S405 includes the following steps: S4051, Construction Phase t Small sample set for target detection in autonomous driving Among them, the small sample set for autonomous driving target detection Including all known categories in the first t stages One sample; S4052, in In this phase, after completing incremental learning of autonomous driving object detection using the updated autonomous driving object detection training set, the autonomous driving object detection small sample set is then used... Fine-tuning and training of the open target detection model for the new stage.

5. The open-world object detection method based on adaptive semantic degradation learning according to claim 3, characterized in that, Step S5 includes the following steps: S501. Iteratively execute steps S1-S4 to complete the training of the open target detection model; S502, Acquire images of the autonomous driving scene to be detected; S503. Based on the autonomous driving scene image to be detected, multi-scale image features are obtained using the pyramid extractor in the trained open object detection model. S504. Encode multi-scale image features to obtain high-level semantic features; S505. Input the high-level semantic features into the known category branch to obtain the known class results, and perform semantic downgrading on the known classes to obtain cluster-level classification; S506. Perform target detection on known class results and cluster-level results, and delete target detection results that are greater than a preset threshold to obtain the detection of unknown classes, thus completing the detection of targets in the open world of autonomous driving scenarios.

Images