Open world target detection category discovery method based on semi-supervised contrast clustering
By adopting semi-supervised contrast clustering model in the open world, the problem that existing technology cannot independently discover and learn new objects is solved, and accurate detection and learning of new object categories is achieved to adapt to the needs of dynamic environments.
Patent Information
- Application Number
- CN202411521021.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-25
AI Technical Summary
Existing object detection methods cannot independently discover and learn new objects in the open world, and can only learn from artificially marked data sets, and cannot adapt to dynamically changing object categories.
The open world object detection category discovery method based on semi-supervised contrast clustering is adopted. By obtaining the original and new data sets of the open world, a semi-supervised contrast clustering model is constructed, and the detection and recognition of known categories is realized, and pseudo-labels are generated through clustering filters, and incremental training is performed to identify new categories.
It realizes accurate detection and learning of new object categories in the open world, breaks through the limitations of traditional detection systems, can adapt to emerging object categories in dynamic environments, and improves the ability to understand unknown objects.
Smart Images
Figure CN119516302A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of fully automated products and relates to an open-world target detection category discovery method based on semi-supervised contrast clustering. Background Art
[0002] In recent years, with the deepening of research in the field of object detection, people can discover and learn new concepts by observing the world, but existing methods can only learn from manually annotated datasets and cannot autonomously discover and learn new types of objects. Especially under the closed world assumption, object categories are predefined. However, the real world is open and dynamic, object categories are not static, and new categories are constantly emerging. Therefore, traditional object detection methods are powerless when dealing with these unknown categories. This limitation motivates us to explore methods that can detect and learn new object categories in the open world.
[0003] In addition, existing methods can only learn from existing labeled data sets. The types of objects in the open world are diverse and disorganized, and they cannot learn unknown objects in unknown environments. Summary of the invention
[0004] In order to solve the above problems, the technical solution adopted by the present invention is: a method for open world object detection category discovery based on semi-supervised contrast clustering, comprising the following steps:
[0005] Obtain open world original datasets and new open world datasets;
[0006] Construct an open-world object detection category model based on semi-supervised contrastive clustering to detect and identify known categories of open-world objects, as well as detect, cluster and identify unknown categories;
[0007] Based on the training data in the open world original data set and the training data in the open world new data set, the open world object detection category model based on semi-supervised contrast clustering is trained to obtain a trained open world object detection category model based on semi-supervised contrast clustering;
[0008] The test set data in the new open world dataset is input into the trained open world object detection category model based on semi-supervised contrastive clustering to realize the recognition of new open world object categories.
[0009] Further: the open world object detection category model of semi-supervised contrastive clustering includes:
[0010] The open object detector model is used to identify objects in the open world raw dataset, classify the identified objects into known categories, and classify the unidentified objects into unknown categories;
[0011] Semi-supervised comparative clustering model: used to perform instance-level and cluster-level clustering in a semi-supervised manner based on the image data of unrecognized objects in unknown categories and the predicted image data of known objects output by the open object detector model, so as to achieve clustering of object targets in unknown categories;
[0012] Clustering filter module: used to select better clustering clusters for the object targets in the unknown category output by the semi-supervised comparative clustering model through the silhouette coefficient, and generate pseudo labels for the better clustering clusters;
[0013] New Class Incremental Training Detector: It is used to combine the unknown clusters with good performance and the generated pseudo labels output by the clustering filter module with the known trained categories of the open world original dataset to form a new dataset and deploy it to the open object detector model.
[0014] Further: the image data of unrecognized objects in the unknown category and the predicted known object image data output by the open object detector model are clustered at the instance level and cluster level to achieve the process of clustering the object targets in the unknown category as follows:
[0015] The enhanced image is fed into the ResNet34 network for feature extraction, and the highest-level features are used for clustering;
[0016] According to the contrastive clustering method, a semi-supervised clustering process is implemented using the labels of known objects. Some data are given real labels, and some data labels are set to unknown, which improves the quality of semantic representation. At the same time, the similarity between positive pairs is maximized and the similarity between negative pairs is minimized. The object contrast learning obtained by two independent multi-layer perceptrons is decoupled into two independent subspaces at the instance level and the cluster level, realizing the process of clustering object targets in unknown categories.
[0017] Further: According to the contrastive clustering method, the semi-supervised clustering process is implemented using the labels of known objects, part of the data is given real labels, and part of the data labels are set to unknown, so as to improve the quality of semantic representation, while maximizing the similarity between positive pairs and minimizing the similarity between negative pairs, and decoupling the object contrastive learning obtained by two independent multi-layer perceptrons into two independent subspaces at the instance level and the cluster level, so as to realize the clustering process of the object targets in the unknown category as follows:
[0018] The feature F extracted by the backbone network ResNet34 is used for the feature enhanced image, and a double-layer multi-layer perceptron g is used I Obtaining feature representation of physical objects: I i =g I (F i ),
[0019] At the same time, the cosine distance is used to calculate The pairwise similarity of is as follows:
[0020]
[0021] Where t1,t2∈{a,b},i,j∈[1,N]; N represents the number of samples in a given batch;
[0022] Optimizing similarity without loss of generality, the contrast loss for a given sample X is defined as follows:
[0023]
[0024] Where: {k +} is the same as A set of pairs of instances; {k -} is the same as The set of negative pairs of instances; T I is an instance-level temperature parameter used to control softness. The instance-level contrast loss is calculated as follows:
[0025]
[0026] Using another two-layer perceptron MLP (g C ), through C i =g G (F i ) projects the image vector into an M-dimensional space, where M is the number of clusters, given an enhanced batch input Output C a =g C (F a ) should be an N×M matrix;
[0027] Only the similarities between different clusters are considered. It is understood as the representation of the mth cluster and the ath augmented input, which will be combined with the rest of C a and C b The columns form 2M-1 pairs, only and This pair is considered a positive sample pair, and all the remaining pairs are negative pairs. The cosine distance is used to calculate the two clusters. The pairwise similarity of m,n∈[1,M] is used to reduce the distance between two enhanced representations of the same cluster. The calculation formula is as follows:
[0028]
[0029] T C is the cluster-level temperature parameter, is with A cluster set of negative sample pairs is formed, τ represents the transpose of the matrix, where t1, t2∈{a, b}, represents the representation of the same sample after different feature enhancement;
[0030] The final cluster-level loss function is defined as follows:
[0031]
[0032] in, It is the entropy of the probability of cluster assignment under each feature enhancement representation in the batch;
[0033] M clusters are obtained from all known and unknown objects. Assuming there are K known objects, there will be K corresponding clusters. Therefore, the basic true values of the known objects are used to screen out clusters that match each known object.
[0034] Further: the process of using the basic truth value of the known object to screen out the clusters matching each known object is as follows:
[0035] Assume P i = argmax(C i ) is the sample X i The predicted clustering, and the number of known objects is M,
[0036] Then, the confusion matrix is calculated on the predicted clusters and the true value labels of the known objects The cost matrix is calculated as follows:
[0037]
[0038] Find the best one-to-one matching between M clusters and M labels by minimizing the cost matrix:
[0039]
[0040] P is the set of all possible one-to-one matches, where in the best match M, if the i-th cluster matches the j-th known object label, then that cluster will be ignored when generating pseudo-labels.
[0041] Further: the process of selecting a better clustering cluster by using the silhouette coefficient for the object target in the unknown category output by the semi-supervised contrast clustering model and generating a pseudo label for the better clustering cluster is as follows:
[0042] The preliminary clustering results are evaluated through evaluation indicators to determine the degree of match between the clusters and the true labels;
[0043] The silhouette coefficient is used to test clusters, quantify the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and select clusters with high silhouette coefficient scores, so that they are closer to the cluster center and farther away from other clusters, thereby screening out high-quality clusters; the silhouette coefficient value range is (-1, +1), a high value indicates good clustering quality, and a low value indicates over-clustering or under-clustering, and clusters with better scores will be selected from high to low;
[0044] The selected high-quality clusters are then quantitatively filtered, and clusters with less than 100 samples are set as clusters below the threshold. Clusters with too low sample numbers are eliminated, and then unknown clusters with excellent performance are identified, and pseudo labels are generated for these clusters.
[0045] Furthermore: the evaluation indicators include ACC clustering accuracy, NMI normalized mutual information and Purity purity.
[0046] Further: the process of combining the unknown clusters with excellent performance output by the clustering filter module and the generated pseudo labels with the categories of the unknown clusters with excellent performance output by the open world original data set to form a new data set and deploy it to the open object detector model is as follows:
[0047] Filter out clusters whose samples are smaller than a specific threshold r;
[0048] The silhouette coefficient is selected to measure the quality of the clustering results, all clusters are sorted according to the silhouette coefficient, and pseudo labels are assigned to the top k clusters;
[0049] Finally, the new k concepts and previously known objects form a new dataset for training open-set object detectors.
[0050] Further: the process of using the silhouette coefficient to test clusters, quantifying the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and selecting clusters with high silhouette coefficient scores, making them closer to their cluster center and away from other clusters, and thus screening out high-quality clusters is as follows:
[0051] Assuming that a sample X belongs to the mth cluster, the calculation formula of the silhouette coefficient of X is:
[0052]
[0053] Where a(X i ) is X i Belong to the same cluster C m The average distance of other samples; b(X i ) is the minimum distance between X and other clusters:
[0054]
[0055] Among them, dist(X i ,X j ) is the cluster space X i With X j The square root of the distance between the two, the silhouette coefficient of a cluster is defined as the average silhouette coefficient of all samples belonging to the cluster:
[0056]
[0057] The silhouette coefficient ranges from -1 to +1, with high values indicating good clustering quality and low values indicating over-clustering or under-clustering.
[0058] The present invention provides an open-world object detection category discovery method based on semi-supervised contrastive clustering, which has the following advantages:
[0059] An innovative open-world object detection method is proposed, which utilizes an open set of detectors to achieve accurate recognition of known objects while effectively detecting potential unknown objects. The method of this application breaks through the limitations of traditional detection systems and can adapt to newly emerging object categories in dynamic environments.
[0060] A state-of-the-art semi-supervised contrastive clustering method is proposed, which aims to efficiently group unknown objects. This method combines the label information of known objects to supervise the clustering process, thereby improving clustering accuracy and promoting the discovery of new concepts. This strategy not only improves the ability to understand unknown objects, but also provides a reliable foundation for subsequent detection tasks. Semi-supervised clustering methods do not require manual labeling. Through this method, the same samples can be found from the unknown and marked as a class, which greatly reduces the manual workload and improves the efficiency of category discovery in open-world object detection.
[0061] A cluster filtering method is proposed to generate pseudo labels for highly reliable clustering results. The method of this application allows newly identified categories to be learned together with previously known objects, thereby optimizing the detector retraining process. This innovation enables the detector to continuously adapt to new environments and achieve accurate detection of new object categories. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0063] Figure 1 The following is an overall flow chart of the open world object detection training process of the present invention;
[0064] Figure 2 This is the structure diagram of the open target detector model;
[0065] Figure 3 It is a detailed structural diagram of the semi-supervised comparative clustering module of the present invention;
[0066] Figure 4 It is a detailed structural diagram of the clustering filter module of the present invention;
[0067] Figure 5 Detect known and unknown objects for open world targets;
[0068] Figure 6 Optimize cluster indicator filtering visualization bar chart for clustering filters;
[0069] Figure 7 A graph showing the effect of new categories obtained by clustering the KITTI dataset. DETAILED DESCRIPTION
[0070] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0072] A method for open-world object detection category discovery based on semi-supervised contrastive clustering, comprising the following steps:
[0073] S1: Obtain the open world original dataset and the open world new dataset;
[0074] S2: Build an open-world object detection category model based on semi-supervised contrastive clustering to detect and identify known categories of open-world objects, and detect, cluster and identify unknown categories;
[0075] S3: Based on the training data in the open world original data set and the training data in the open world new data set, the open world object detection category model based on semi-supervised contrast clustering is trained to obtain a trained open world object detection category model based on semi-supervised contrast clustering;
[0076] S4: Input the test set data in the new open world dataset into the trained open world object detection category model based on semi-supervised contrast clustering to realize the recognition of new open world object categories.
[0077] The steps S1 / S2 / S3 / S4 are performed sequentially;
[0078] Figure 1 The following is an overall flow chart of the training process of the open world object detection model of the present invention;
[0079] Further: the open world object detection category model of semi-supervised contrastive clustering includes:
[0080] The open object detector model is used to identify objects in the open world raw dataset, classify the identified objects into known categories, and classify the unidentified objects into unknown categories;
[0081] Semi-supervised comparative clustering model: used to perform instance-level and cluster-level clustering in a semi-supervised manner based on the image data of unrecognized objects in unknown categories and the predicted image data of known objects output by the open object detector model, so as to achieve clustering of object targets in unknown categories;
[0082] Clustering filter module: used to select better clustering clusters for the object targets in the unknown category output by the semi-supervised comparative clustering model through the silhouette coefficient, and generate pseudo labels for the better clustering clusters;
[0083] New Class Incremental Training Detector: It is used to combine the unknown clusters with good performance and the generated pseudo labels output by the clustering filter module with the known trained categories of the open world original dataset to form a new dataset and deploy it to the open object detector model.
[0084] Figure 2 This is the structure diagram of the open target detector model;
[0085] The open object detector model is a detector based on the Faster R-CNN (ultra-fast regional convolutional neural network) model, which is divided into three parts: ResNet50 (50-layer deep convolutional neural network) feature extraction module, unknown perception RPN (region proposal network) module and ROI Head (region of interest) head module. It can detect known objects and identify unknown objects by calculating energy scores.
[0086] Figure 3 It is a detailed structural diagram of the semi-supervised comparative clustering module of the present invention;
[0087] The process of clustering the unrecognized object image data in the unknown category and the predicted known object image data output by the open object detector model and implementing clustering of the object targets in the unknown category is as follows:
[0088] S21: The enhanced image is input into the ResNet34 (34-layer deep convolutional neural network) network for feature extraction, and the highest level features will be used for clustering;
[0089] S22: According to the contrastive clustering method, a semi-supervised clustering process is implemented using the labels of known objects. Some data are given real labels, and some data labels are set to unknown, to improve the quality of semantic representation. At the same time, the similarity between positive pairs is maximized and the similarity between negative pairs is minimized. The object contrast learning obtained by two independent multi-layer perceptrons is decoupled into two independent subspaces at the instance level and the cluster level, to realize the process of clustering object targets in unknown categories.
[0090] The data of the detected unknown objects and predicted known objects are enhanced by applying two different enhancement methods, as follows:
[0091] The detected unknown objects (without true value labels) and the predicted known objects (with true value labels) are used as data input, and contrast clustering is performed in a semi-supervised manner. At the same time, the contrast clustering algorithm defines the enhanced image pairs from the same instance as positive sample pairs, and defines other image pairs as negative sample pairs. ;
[0092] The input data is feature enhanced. Given a batch size of N samples, in {X1,…,X N After data increment processing for each instance of}, 2N samples can be obtained Among them, the augmented image pairs of the same instance will be regarded as positive pairs, e.g. and Secondly, if two samples X i and X j , have the same label, then and will be considered as positive sample pairs. In addition, the pairing between the two augmented batches and will also be considered a positive pair. or If the above conditions are not met, it is considered a negative pair. i , only the enhanced image pair Only the two images are considered as positive image pairs, and all other image pairs are negative image pairs.
[0093] According to the contrastive clustering method, the semi-supervised clustering process is implemented using the labels of known objects, some data are given real labels, and some data labels are set as unknown, so as to improve the quality of semantic representation, maximize the similarity between positive pairs, minimize the similarity between negative pairs, and decouple the object contrastive learning obtained by two independent multi-layer perceptrons into two independent subspaces at the instance level and the cluster level. The process of clustering the object targets in the unknown category is as follows:
[0094] S221: The feature F extracted by the backbone network ResNet34 is used to extract the feature F of the image after feature enhancement, and a double-layer multilayer perceptron g is used. I Obtaining the feature representation of the physical object: I i =g I (F i ),
[0095] At the same time, the cosine distance is used to calculate The pairwise similarity of is as follows:
[0096]
[0097] Where t1,t2∈{a,b},i,j∈[1,N]; N represents the number of samples in a given batch;
[0098] S222: In order to optimize the similarity without loss of generality, the contrast loss for a given sample X is defined as follows:
[0099]
[0100] {k +} is the same as A set of pairs of instances; {k -} is the same as The set of negative pairs of instances; T I is the instance-level temperature parameter used to control softness. In the experiment, T I Set to 1.5; the instance-level contrast loss is calculated as follows:
[0101]
[0102] S223: Cluster-level contrastive clustering is similar to instance-level contrastive learning, in that both known objects and unknown objects are used for cluster-level contrastive clustering. However, the true value of the known object is only involved in post-processing, that is, clusters matching the known object category are ignored. Using another two-layer MLP (g C ), through C i =g G (F i ) projects the image vector into an M-dimensional space, where M is the number of clusters, given an enhanced batch input Output Ca =g C (F a ) should be an N×M matrix; MLP is a multilayer perceptron, which is a feedforward neural network consisting of an input layer, a hidden layer, and an output layer.
[0103] Two-layer MLP means the number of hidden layers is 2, i.e. there are two hidden layers.
[0104] S224: Only consider the similarity between different clusters, It is understood as the representation of the mth cluster and the ath augmented input, which will be combined with the rest of C a and C b The columns form 2M-1 pairs, only and This pair is considered a positive sample pair, and all the remaining pairs are negative pairs. The cosine distance is used to calculate the two clusters. The pairwise similarity of m,n∈[1,M] is used to reduce the distance between two enhanced representations of the same cluster. The calculation formula is as follows:
[0105]
[0106] T C is the cluster-level temperature parameter, is with A clustering set of negative sample pairs is formed, τ represents the transpose of the matrix, where t1, t2∈{a, b}, represents the representation of the same sample after different feature enhancement.
[0107] The final cluster-level loss function is defined as follows:
[0108]
[0109] Where H(C) is the entropy of the cluster assignment probability under each feature augmentation representation in the batch. It encourages the model to avoid assigning most instances to one cluster by penalizing high levels of cluster assignment uncertainty.
[0110] S225: After the training, M clusters are obtained from all known and unknown objects. Assuming there are K known objects, there will be K corresponding clusters. Therefore, the basic true values of the known objects are used to screen out clusters that match each known object.
[0111] Further: the process of using the basic truth value of the known object to screen out the clusters matching each known object is as follows:
[0112] Assume P i = argmax(C i ) is the sample Xi The predicted clusters of the known objects are M, and then the confusion matrix is calculated on the predicted clusters and true value labels of the known objects. The cost matrix is calculated as follows:
[0113]
[0114] Find the best one-to-one matching between M clusters and M labels by minimizing the cost matrix:
[0115]
[0116] P is the set of all possible one-to-one matches, where in the best match M, if the i-th cluster matches the j-th known object label, then that cluster will be ignored when generating pseudo-labels.
[0117] A clustering filter that generates pseudo labels specifically for high-confidence clusters
[0118] Figure 4 It is a detailed structural diagram of the clustering filter module of the present invention;
[0119] Further: the process of selecting a better clustering cluster by using the silhouette coefficient for the object target in the unknown category output by the semi-supervised contrast clustering model and generating a pseudo label for the better clustering cluster is as follows:
[0120] S31: Evaluate the preliminary clustering results to determine the matching degree between the clusters and the true labels through evaluation indicators; the evaluation indicators include ACC clustering accuracy, NMI normalized mutual information and Purity purity;
[0121] S32: Use the silhouette coefficient to test clusters, quantify the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and select clusters with high silhouette coefficient scores, so that they are closer to their cluster center and away from other clusters, thereby screening out high-quality clusters; the silhouette coefficient range is (-1, +1), high values indicate good clustering quality, and low values indicate over-clustering or under-clustering. We will select clusters with better scores from high to low.
[0122] S33: Then perform quantitative filtering on the selected high-quality clusters, set the clusters with less than 100 samples as below-threshold clusters, and remove clusters with too low sample numbers to ensure that only unknown clusters with statistical significance and robustness are retained. Ultimately, through these screening steps, unknown clusters with excellent performance can be efficiently identified, and pseudo labels can be generated for these clusters for further open set object detector model training and optimization.
[0123] Since the clusters generated by the semi-supervised contrastive clustering model cannot be directly assigned to new pseudo labels to retrain the detector. We find that: (a) some clusters may contain only a small number of samples; (b) some clusters cover different kinds of objects. Both of these clusters should not be retrained as new concepts. Therefore, we design a cluster filter to generate pseudo labels only for high-consistency clusters.
[0124] The process of combining the unknown clusters with excellent performance output by the clustering filter module and the generated pseudo labels with the categories of the unknown clusters with excellent performance output by the open world original data set to form a new data set and deploy it to the open object detector model is as follows:
[0125] S41: Filter out clusters whose samples are smaller than a specific threshold r;
[0126] S42: Select the silhouette coefficient to measure the quality of the clustering results, sort all clusters according to the silhouette coefficient, and assign pseudo labels to the top k clusters;
[0127] S43: Finally, the new k concepts and previously known objects form a new dataset for training open-set object detectors.
[0128] The process of using the silhouette coefficient to test clusters, quantifying the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and selecting clusters with high silhouette coefficient scores, making them closer to their cluster center and away from other clusters, and thus screening out high-quality clusters is as follows:
[0129] Assuming that a sample X belongs to the mth cluster, the calculation formula of the silhouette coefficient of X is:
[0130]
[0131] Where a(X i ) is X i Belong to the same cluster C m The average distance of other samples; b(X i ) is the minimum distance between X and other clusters:
[0132]
[0133] Among them, dist(X i ,X j ) is the cluster space X i With X j The square root of the distance between the two, the silhouette coefficient of a cluster is defined as the average silhouette coefficient of all samples belonging to the cluster:
[0134]
[0135] The silhouette coefficient ranges from -1 to +1, with high values indicating good clustering quality and low values indicating over-clustering or under-clustering.
[0136] This embodiment 1: It is built by a GTX 3090 graphics card server and an Nvidia Jetson Xavier mobile lightweight computer as the test running environment of the method of the present invention. All experiments are carried out on the server for model training, and the deployment model is tested on the Nvidia Jetson Xavier. The specific hardware configuration information is shown in Table 1.
[0137] Table 1 Server hardware configuration
[0138] Configuration Specification CPU Intel i7-10700K CPU @ 3.8GHz GPU GTX3090 Memory 32GB harddisk 500GB+6TB system Ubuntu 18.04 GPU GTX3090
[0139] Table 2 Nvidia Jetson Xavier hardware configuration
[0140]
[0141]
[0142] This embodiment uses Pycharm as the development environment of the method of the present invention, and Python as the programming language to complete the method design and development. The software environment for running the method in this embodiment includes: operating system Ubuntu 16.04.01, Python3.8. The specific software environment is shown in Table 2.
[0143] Table 3 Software environment
[0144] Software Environment Version operating system Ubuntu 16.04.01 Python Version 3.8 Development Environment Pycharm language Python
[0145] Table 4 Experimental dataset statistics
[0146]
[0147] Open set object detection:
[0148] (1) We collect image data (without true value labels) in open world scenes through existing datasets (with true value labels) or image acquisition devices.
[0149] (2) We extract features from the original image data through the backbone network ResNet50 module and retain the most accurate features of the last layer.
[0150] (3) We use the unknown perception module RPN to generate and predict candidate boxes based on object features, and continuously train and optimize the objectness score of the object to enable the model to better distinguish foreground objects from the background.
[0151] (4) We use the region of interest head module (ROI Head) to regress the box boundaries and predict the box object category of the candidate region generated by RPN.
[0152] (5) We use the energy-based unknown recognition model to fit the energy values of known objects and the energy values of unknown objects to form a WeiBull distribution. We calculate the energy value of the newly recognized object and project it into the energy function to determine whether it should be classified as known or unknown.
[0153] Based on the semi-supervised contrastive clustering model, the image data of unrecognized objects in the unknown category output by the open object detector model and the predicted image data of known objects are clustered at the instance level and the cluster level to achieve clustering of object targets in the unknown category;
[0154] To test the ability to mine new categories in unknown objects, we combined PascolVOC2007 (containing 20 categories) and MS-COCO datasets (containing 60 categories) to create a custom data set. We set different classification and learning tasks, each task learns 20 new categories, train an open set detector through multiple rounds of tasks, and use it to identify known objects and find all unknown objects. Then, the detected known objects and unknown objects are collected as a custom dataset for new category mining.
[0155] Table 1
[0156]
[0157] Table 1 shows that our semi-supervised comparative clustering method outperforms the other four baseline methods by a large margin in all three tasks. This is because all other methods are unsupervised. In contrast, we modify our clustering method to be semi-supervised, which learns representations not only from augmented pairs but also from objects belonging to the same category. This method produces a better clustering semantic space than the other methods. In Task 1 and Task 2, our method outperforms the four baseline methods, while in Task 3, our method is slightly lower in purity statistics by DTC+0.5 percentage points, and outperforms the method in all other indicators.
[0158] Table 2
[0159]
[0160] We compare our semi-supervised contrastive clustering module with the baseline method, the unsupervised contrastive clustering (CC) algorithm, in Table 2. Since the ground-truth labels of known objects are not used in CC, the known objects are ignored when calculating the metrics. As can be seen from the clustering metrics, our semi-supervised contrastive clustering method scores higher than the contrastive clustering algorithm in all tasks. This shows that the use of the ground-truth labels of known objects in a semi-supervised manner in instance-level clustering helps to form better semantic representations, thereby facilitating the clustering process.
[0161] Figure 5 The following are some of the detection results of the open set object detector. From the top two rows of the figure, we can see that we have achieved good results in the recognition and discovery of unknown objects. After completing Task 1, the model can recognize 20 categories. The model can effectively detect unknown objects that have not been trained. The last row of the figure shows the better clustering results obtained after re-clustering the unknown objects.
[0162] Figure 6 A visualization of the first five clusters found by cluster filtering for our proposed open-world object detection model. After low-number filtering and data analysis, it can be seen that the group with the highest SI value also has the highest purity value, and the purity value decreases as the SI value decreases. The SI and purity change in roughly the same trend. In this experiment, our method found a cluster with a purity of 100%. However, there are only five samples in this cluster, so it will be regarded as an over-clustering result and filtered out in pseudo generation. The first two clusters with a purity of 94.7% and 942 samples will be marked as new categories.
[0163] Figure 7 This is a visualization of the new category clustering using the OWDet method on the KITTI autonomous driving dataset in an open world scenario. It can be seen that better clusters can effectively cluster road signs among many unknown objects to form new label data.
[0164] After a large number of experiments and result data analysis, the open-world object detector OWDet proposed by us is superior to the Faster R-CNN model detector in the process of identifying known and detecting unknown objects. At the same time, we conducted experiments on various datasets such as CIFAR, STL, ImageNet, etc. After semi-supervised comparative clustering of the identified unknown objects, the ACC, NMI, and Purity obtained by analyzing the clustering results are superior to traditional unsupervised clustering algorithms Kmeans, Kmeans++ and deep transfer clustering algorithm DTC, and the efficient parameter-free clustering algorithm FINCH based on first-order neighbor relations. At the same time, in the classic open-world object detection datasets PascolVoc, COCO and autonomous driving Kitti datasets, 94.7% purity was obtained in 942 cluster samples. In the incremental learning process, the indicators of the previously known classes did not drop significantly, proving that the semi-supervised comparative clustering algorithm we proposed is efficient and robust.
[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for open-world object detection category discovery based on semi-supervised contrastive clustering, characterized by: The following steps are involved: Obtain open world original datasets and new open world datasets; Construct an open-world object detection category model based on semi-supervised contrastive clustering to detect and identify known categories of open-world objects, as well as detect, cluster and identify unknown categories; Based on the training data in the open world original data set and the training data in the open world new data set, the open world object detection category model based on semi-supervised contrastive clustering is trained to obtain a trained open world object detection category model based on semi-supervised contrastive clustering; The test set data in the new open world dataset is input into the trained open world object detection category model based on semi-supervised contrastive clustering to realize the recognition of new open world object categories.
2. The open-world object detection category discovery method based on semi-supervised contrastive clustering according to claim 1, characterized in that: The open-world object detection category model of semi-supervised contrastive clustering includes: The open object detector model is used to identify objects in the open world raw dataset, classify the identified objects into known categories, and classify the unidentified objects into unknown categories; Semi-supervised comparative clustering model: used to perform instance-level and cluster-level clustering in a semi-supervised manner based on the image data of unrecognized objects in unknown categories and the predicted image data of known objects output by the open object detector model, so as to achieve clustering of object targets in unknown categories; Clustering filter module: used to select better clustering clusters for the object targets in the unknown category output by the semi-supervised comparative clustering model through the silhouette coefficient, and generate pseudo labels for the better clustering clusters; New Class Incremental Training Detector: It is used to combine the unknown clusters with good performance and the generated pseudo labels output by the clustering filter module with the known trained categories of the open world original dataset to form a new dataset and deploy it to the open object detector model.
3. The open-world object detection category discovery method based on semi-supervised contrastive clustering according to claim 2, characterized in that: The process of performing instance-level and cluster-level clustering on the image data of unrecognized objects in the unknown category and the predicted known object image data output by the open object detector model to achieve clustering of object targets in the unknown category is as follows: The enhanced image is fed into the ResNet34 network for feature extraction, and the highest-level features are used for clustering; According to the contrastive clustering method, a semi-supervised clustering process is implemented using the labels of known objects. Some data are given real labels, and some data labels are set to unknown, which improves the quality of semantic representation. At the same time, the similarity between positive pairs is maximized and the similarity between negative pairs is minimized. The object contrast learning obtained by two independent multi-layer perceptrons is decoupled into two independent subspaces at the instance level and the cluster level, realizing the process of clustering object targets in unknown categories.
4. The method for open-world object detection category discovery based on semi-supervised contrastive clustering according to claim 3, characterized in that: According to the contrastive clustering method, the semi-supervised clustering process is implemented using the labels of known objects, some data are given real labels, and some data labels are set as unknown, so as to improve the quality of semantic representation, maximize the similarity between positive pairs, minimize the similarity between negative pairs, and decouple the object contrastive learning obtained by two independent multi-layer perceptrons into two independent subspaces at the instance level and the cluster level. The process of clustering the object targets in the unknown category is as follows: The feature F extracted by the backbone network ResNet34 is used for the feature enhanced image, and a double-layer multi-layer perceptron g is used I Obtaining feature representation of physical objects: I i =g I (F i ), At the same time, the cosine distance is used to calculate The pairwise similarity of is as follows: Where t1, t2∈{a, b}, i, j∈[1, N]; N represents the number of samples in a given batch; Optimizing similarity without loss of generality, the contrast loss for a given sample X is defined as follows: Where: {k + } is the same as A set of pairs of instances; {k - } is the same as The set of negative pairs of instances; T I is an instance-level temperature parameter used to control softness. The instance-level contrast loss is calculated as follows: Using another two-layer perceptron MLP (g C ), through C i =g G (F i ) projects the image vector into an M-dimensional space, where M is the number of clusters, given an enhanced batch input Output C a =g C (F a ) should be an N×M matrix; Only the similarities between different clusters are considered. It is understood as the representation of the mth cluster and the ath augmented input, which will be combined with the rest of C a and C b The columns form 2M-1 pairs, only and This pair is considered a positive sample pair, and all the remaining pairs are negative pairs. The cosine distance is used to calculate the two clusters. The pairwise similarity of, where m, n ∈ [1, M], the following clustering, contrast loss is used to narrow the distance between two enhanced representations of the same cluster. The calculation formula is as follows: T C is the cluster-level temperature parameter, is with Form a cluster set of negative sample pairs, τ represents the transpose of the matrix, where t1, t2∈{a, b}, represents the representation of the same sample after different feature enhancement; The final cluster-level loss function is defined as follows: in, It is the entropy of the probability of cluster assignment under each feature enhancement representation in the batch; M clusters are obtained from all known and unknown objects. Assuming there are K known objects, there will be K corresponding clusters. Therefore, the basic true values of the known objects are used to screen out clusters that match each known object.
5. The method for open-world object detection category discovery based on semi-supervised contrastive clustering according to claim 4, characterized in that: The process of using the ground truth values of known objects to screen out clusters matching each known object is as follows: Assume P i = argmax(C i ) is the sample X i The predicted clusters of the known objects are M, and then the confusion matrix is calculated on the predicted clusters and true value labels of the known objects. The cost matrix is calculated as follows: Find the best one-to-one matching between M clusters and M labels by minimizing the cost matrix: P is the set of all possible one-to-one matches, where in the best match M, if the i-th cluster matches the j-th known object label, then that cluster will be ignored when generating pseudo-labels.
6. The open-world object detection category discovery method based on semi-supervised contrastive clustering according to claim 2, characterized in that: The process of selecting a better clustering cluster by using the silhouette coefficient for the object target in the unknown category output by the semi-supervised contrast clustering model and generating a pseudo label for the better clustering cluster is as follows: The preliminary clustering results are evaluated through evaluation indicators to determine the degree of match between the clusters and the true labels; The silhouette coefficient is used to test clusters, quantify the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and select clusters with high silhouette coefficient scores, so that they are closer to the center of their cluster and farther away from other clusters, thereby screening out high-quality clusters; the silhouette coefficient value range is (-1, +1), a high value indicates good clustering quality, and a low value indicates over-clustering or under-clustering, and clusters with better scores will be selected from high to low; The selected high-quality clusters are then quantitatively filtered, and clusters with less than 100 samples are set as clusters below the threshold. Clusters with too low sample numbers are eliminated, and then unknown clusters with excellent performance are identified, and pseudo labels are generated for these clusters.
7. The method for open-world object detection category discovery based on semi-supervised contrastive clustering according to claim 6, characterized in that: The evaluation indicators include ACC clustering accuracy, NMI normalized mutual information and Purity purity.
8. The method for open-world object detection category discovery based on semi-supervised contrastive clustering according to claim 2, characterized in that: The process of combining the unknown clusters with excellent performance output by the clustering filter module and the generated pseudo labels with the categories of the unknown clusters with excellent performance output by the open world original data set to form a new data set and deploy it to the open object detector model is as follows: Filter out clusters whose samples are smaller than a specific threshold r; The silhouette coefficient is selected to measure the quality of the clustering results, all clusters are sorted according to the silhouette coefficient, and pseudo labels are assigned to the top k clusters; Finally, the new k concepts and previously known objects form a new dataset for training open-set object detectors.
9. The method for open-world object detection category discovery based on semi-supervised contrastive clustering according to claim 5, characterized in that: The process of using the silhouette coefficient to test clusters, quantifying the similarity difference between each cluster sample and its cluster and the nearest neighbor cluster, and selecting clusters with high silhouette coefficient scores, making them closer to their cluster center and away from other clusters, and thus screening out high-quality clusters is as follows: Assuming that a sample X belongs to the mth cluster, the calculation formula of the silhouette coefficient of X is: Where a(X i ) is X i Belong to the same cluster C m The average distance of other samples; b(X i ) is the minimum distance between X and other clusters: Among them, dist(X i , X j ) is the cluster space X i With X j The square root of the distance between the two, the silhouette coefficient of a cluster is defined as the average silhouette coefficient of all samples belonging to the cluster: The silhouette coefficient ranges from -1 to +1, with high values indicating good clustering quality and low values indicating over-clustering or under-clustering.