Open world target detection method based on contrast clustering and multivariate Gaussian distribution

Abstract

The invention discloses an open world target detection method based on contrast clustering and multivariate Gaussian distribution. The method comprises the following steps: training an open world target detector model; performing contrast clustering learning on objects which are not recognized by the open world target detector model based on a semi-supervised contrast clustering model to obtain a contrast clustering result; on the basis of an incremental learning mode, a heap with the best clustering is selected for comparison clustering results output by the semi-supervised comparison clustering model, false labels are marked, the heap is returned in a sample playback mode, and training of the open world target detector model is returned; when the training of the open world target detector model reaches a fixed number of iterations, obtaining a trained open world target detector model; inputting a to-be-detected image into the trained open world target detector model to realize category identification of the to-be-detected image; according to the method, strong / weak data enhancement is carried out on data, and clustered clusters are filtered.

Description

Technical Field

[0001] The present invention belongs to the field of object detection, and relates to an open-world object detection method based on contrast clustering and multivariate Gaussian distribution. Background Art

[0002] In recent years, the problem that robots can only learn from manually annotated datasets and cannot autonomously discover and learn new types of objects has emerged. Under the closed-world assumption, object categories are predefined by the training set. However, the real world contains more objects outside the training set, and new categories are constantly emerging, but traditional object detection methods cannot handle and detect new objects outside the training set. Therefore, people have begun to explore open-world object detection that can detect new objects outside the training set and enable the detector to have the ability to learn these new objects. Summary of the Invention

[0003] In order to solve the above problems, the technical solution adopted by the present invention is: an open-world object detection method based on contrast clustering and multivariate Gaussian distribution, including the following steps:

[0004] Obtain the image to be detected;

[0005] Construct an open-world object detector model for detecting known and unknown objects in the image;

[0006] Train the open-world object detector model;

[0007] Perform contrast clustering learning on the objects not recognized by the open-world object detector model based on a semi-supervised contrast clustering model to obtain a contrast clustering result;

[0008] Based on an incremental learning method, select the heap with the best clustering for the contrast clustering result output by the semi-supervised contrast clustering model, assign pseudo-labels, and return it through the method of sample replay, and return to train the open-world object detector model;

[0009] When the training of the open-world object detector model reaches a fixed number of iterations, obtain the trained open-world object detector model;

[0010] Input the image to be detected into the trained open-world object detector model to realize the category recognition of the image to be detected.

[0011] Furthermore: the open-world object detector model includes an object localization module for detecting the position of the target in the image;

[0012] The object localization module includes a first feature extraction network for extracting the features of the target in the image;

[0013] Object candidate network module: used to predict the centrality of the target in the image and generate candidate bounding boxes;

[0014] Two-stage regression network: Based on the features extracted by the first feature extraction network and the predicted candidate bounding boxes output by the object candidate network module, extract features from each layer of the feature pyramid, and then perform refined regression of the candidate bounding boxes and prediction of the intersection over union score I.

[0015] Furthermore: The open-world object detector model includes a classification module for classifying the target in the image based on the target position output by the object localization module;

[0016] The classification module includes: a second feature extraction network for extracting the features of the target in the image;

[0017] RoI Align: Based on the features extracted by the second feature extraction network and the bounding boxes output by the object localization module, sample the features output by the second feature extraction network according to the output bounding boxes;

[0018] Pseudo-feature generation module: used to generate pseudo-features based on the sampled features output by RoI Align;

[0019] Classification network: used to match the target in the image with the known categories based on the sampled features output by RoI Align and the pseudo-features output by the pseudo-feature generation module.

[0020] Furthermore: The process of training the open-world object detector model is as follows:

[0021] The first stage: Train the object localization module;

[0022] The second stage: Freeze the network parameters of the object localization module, use the fitting of the multivariate Gaussian distribution, and start training the classification module:

[0023] The third stage: Continue to train the classification module, generate pseudo-features according to the fitted Gaussian distribution, the pseudo-features of each category will be input into the classification network, obtain the classification results and calculate the energy values, and through contrast learning with the normally extracted features, let the classification network learn the boundaries of each category;

[0024] The fourth stage: Distinguish known objects and unknown objects by calculating the threshold.

[0025] Furthermore: The semi-supervised contrast clustering model includes a strong data and weak data augmentation module, a ResNet34 feature extraction module, an instance-level contrast clustering module, and a clustering-level contrast clustering module;

[0026] The strong data and weak data augmentation module is used to obtain a set of strongly augmented and two sets of weakly augmented images based on the unknown objects output by the open-world object detector model and the predicted known objects;

[0027] The ResNet34 feature extraction module is used to extract features from the images augmented by the strong data and weak data augmentation module;

[0028] The instance-level contrast clustering module is used to output instance-level features based on the features extracted by the ResNet34 feature extraction module;

[0029] The clustering-level contrast clustering module is used to output a clustering result based on the instance-level features output by the instance-level contrast clustering module, and then calculate the instance-level and clustering-level similarities to maximize the similarity of positive sample pairs.

[0030] Furthermore: The calculation process of the overall loss of the instance-level contrast clustering module is as follows:

[0031] Use the first double-layer MLP to obtain the feature representation I of the object i , and at the same time use the cosine distance as the similarity, taking the strongly augmented sample and the weakly augmented sample as an example, the specific formula is as follows:

[0032]

[0033] where: t1, t2 ∈ {a, b}, i, j ∈ [1, N], and the contrast loss of the sample combination is defined as follows:

[0034]

[0035] {k +} is the set of instances that form positive pairs with ; {k -} is the set of instances that form negative pairs with , T I is the instance-level temperature parameter used to control softness. Set T I to 1.5, and the calculation method of the instance-level contrast loss is as follows:

[0036]

[0037] The final overall loss at the instance level is composed of the strongly augmented sample and the weakly augmented sample and the weakly augmented sample and the weakly augmented sample together, and the specific formula is as follows.

[0038] L ins = L ins(a,b) + L ins(b,c) (14)

[0039] Furthermore, the calculation process of the overall loss of the clustering-level contrast clustering module is as follows:

[0040] Another two-layer MLP is used to project the features extracted by ResNet34 into an M-dimensional space, where M is the number of clustering clusters, for strongly data-augmented samples and weakly data-augmented samples For example, given an augmented input output an N×M matrix, is the representation of the m-th cluster of the augmented sample a, C a and C b There are a total of 2M columns, ignoring and itself, and This pair is regarded as a positive sample pair, and all the remaining sample pairs are defined as negative sample pairs. The cosine distance is used to calculate the pairwise similarity of the two clusters where m, n ∈ [1, M]. The following clustering loss and contrast loss reduce the distance between the two augmented representations of the same cluster. The calculation formula is as follows:

[0041]

[0042] where t1, t2 ∈ {a, b}, T C is the temperature coefficient, is the clustering set that forms a negative sample pair with The clustering-level loss function is defined as follows:

[0043]

[0044] where, is the entropy of the cluster assignment probability under each feature augmentation representation in the batch. By penalizing the high-level clustering assignment uncertainty, the model is encouraged to avoid assigning most instances to one cluster; The final overall loss at the clustering level is composed of strongly augmented samples and weakly augmented samples weakly augmented samples and weakly augmented samples and strongly augmented samples and weakly augmented samples Specifically, the formula is as follows:

[0045] L clu = L clu(a,b) + L clu(a,c) + L clu(b,c)(18); The final loss of this module is jointly obtained by the instance-level contrast clustering module and the clustering-level contrast clustering module, and the formula is as follows:

[0046] L = L clu + L ins (19).

[0047] An open-world object detection method based on contrast clustering and multivariate Gaussian distribution provided by the present invention. This method uses a multi-stage training method to avoid the influence of classification supervision on object localization, fits the class-conditional multivariate Gaussian distribution to generate pseudo-features, and performs contrast learning on the pseudo-features and the features extracted from the pictures to achieve accurate recognition of known objects and effectively detect potential unknown objects. Our method breaks through the limitations of traditional detection systems and can adapt to newly emerging object categories in a dynamic environment.

[0048] A novel semi-supervised contrast clustering method proposed by this method aims to efficiently group unknown objects. This method performs strong / weak data augmentation on the data and uses the label information of known objects to supervise the clustering process, thereby improving the clustering accuracy. And the clustered clusters are filtered to reduce the cost of manual annotation;

[0049] After a large number of experiments and result data analysis, the open-world object detection method we proposed is superior to the Faster R-CNN model detector in the process of identifying known objects and detecting unknown objects. At the same time, we conducted experiments on various datasets such as Pascal VOC and COCO, and obtained the detection results and clustering results of known objects and unknown objects. It can be analyzed that our detection results are superior to the open-world object detectors ORE, OW-DETR and the open-set object detectors VOS, EnergyScore, etc. The ACC, NMI, and Purity obtained from the clustering results are superior to the traditional unsupervised clustering algorithms Kmeans, the deep transfer clustering algorithm DTC, the efficient parameter-free clustering algorithm FINCH based on first-order neighbor relationships, and the open-set object detection method OSODD that also clusters unknown objects. During the implementation of the incremental learning process, the previous known class metrics did not decrease significantly, proving the high efficiency and good robustness of the semi-supervised contrast clustering algorithm we proposed. Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 Flowchart of the open-world object detection method of the present invention;

[0052] Figure 2 Overall structure diagram of the open-world object detection method of the present invention;

[0053] Figure 3 Structure diagram of the open-world object detector of the present invention;

[0054] Figure 4 Flowchart of the training of the open-world object detector of the present invention;

[0055] Figure 5 Structure diagram of the semi-supervised contrast clustering module of the present invention;

[0056] Figure 6 Effect diagram of detecting known and unknown objects by this method. Detailed implementation manners

[0057] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0058] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The description of at least one exemplary embodiment below is actually only illustrative and in no way restrictive of the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0059] Figure 1 Flowchart of the open-world object detection method of the present invention;

[0060] Figure 2 Overall structure diagram of the open-world object detection method of the present invention;

[0061] An open-world object detection method based on contrast clustering and multivariate Gaussian distribution, comprising the following steps:

[0062] S1: Construct an open-world object detector model for detecting known and unknown objects in an image;

[0063] S2: Train the open-world object detector model;

[0064] S3: Perform contrastive clustering learning on the objects not recognized by the open-world object detector model based on the semi-supervised contrastive clustering model to obtain the contrastive clustering result;

[0065] S4: Based on the incremental learning method, for the contrastive clustering result output by the semi-supervised contrastive clustering model, select the cluster with the best clustering, assign pseudo-labels, and return it through the sample replay method, and return it for training the open-world object detector model;

[0066] S5: When the training of the open-world object detector model reaches a fixed number of iterations, obtain the trained open-world object detector model;

[0067] S6: Input the image to be detected into the trained open-world object detector model to realize the category recognition of the image to be detected.

[0068] The steps S1 / S2 / S3 / S4 / S5 / S6 are executed in sequence;

[0069] The method of the present invention consists of three parts: an open-world object detector that can detect known and unknown objects in an image, a semi-supervised contrastive clustering that clusters unknown objects and assigns pseudo-labels, and incremental learning. In the inference stage of the present invention, the open-world object detector will detect unknown objects and uniformly assign them the label of "unknown object". When the number of unknown objects is large enough, semi-supervised contrastive clustering will be performed. After the semi-supervised contrastive clustering is completed, the unknown objects will be further distinguished, that is, the label of "unknown object" will be output as pseudo-labels such as "unknown object 1" and "unknown object 2". When the number of samples is sufficient, the cluster with the best clustering quality will be filtered out, and the pseudo-labels will be mapped to the objects corresponding to the real world by humans, and then these samples will be used as a new known object training set to retrain the open-world object detector to realize the incremental learning of the open-world object detector.

[0070] Figure 3 It is the structure diagram of the open-world object detector of the present invention;

[0071] Further, the open-world object detector model includes an object localization module for detecting the position of the target in the image;

[0072] The object localization module includes a first feature extraction network for extracting the features of the target in the image;

[0073] The Region Proposal Network (RPN) module of the object candidate network: for predicting the centrality of the target in the image and generating candidate bounding boxes;

[0074] Two-stage regression network: Based on the features extracted by the first feature extraction network and the predicted candidate boxes output by the RPN module, extract features from each layer of the feature pyramid, and then perform refined regression of the candidate boxes and prediction of the intersection over union score I.

[0075] The two-stage regression network includes RoI Align and a regression network (linear layer). The regression network contains two shared linear layers and two independent linear layers. The two independent linear layers are respectively used to regress the box coordinates and predict the intersection over union score mentioned in the text.

[0076] Further: The open-world object detector model includes a classification module for classifying the objects in the image based on the object positions output by the object localization module.

[0077] The classification module includes: a second feature extraction network for extracting the features of the objects in the image.

[0078] RoI Align (Region of Interest Align): Based on the features extracted by the second feature extraction network and the boxes output by the object localization module, sample the features output by the second feature extraction network according to the output boxes.

[0079] Pseudo-feature generation module: Used to generate pseudo-features based on the sampled features output by RoI Align.

[0080] Classification network: Used to match the objects in the image with the known categories based on the sampled features output by RoI Align and the pseudo-features output by the pseudo-feature generation module.

[0081] Figure 4 This is the training flow chart of the open-world object detector of the present invention.

[0082] Further: The process of training the open-world object detector model is as follows:

[0083] The first stage: Train the object localization module.

[0084] The second stage: Freeze the network parameters of the object localization module, fit with a multivariate Gaussian distribution, and start training the classification module. The frozen network parameters mean stopping the parameter update of the frozen network, that is, this part of the model switches from the training mode to the inference mode and does not participate in backpropagation.

[0085] The third stage: Continue to train the classification module, generate features according to the fitted Gaussian distribution. The pseudo-features of each category will be input into the classification network to obtain the classification results and calculate the energy values. Through contrast learning with the normally extracted features, the classification network is made to learn the boundaries of each category.

[0086] Fourth stage: Distinguish known objects and unknown objects by calculating thresholds.

[0087] Furthermore, the process of selecting the best clustering heap from the contrast clustering results output by the semi-supervised contrast clustering model based on the incremental learning method and returning it to the open-world object detector model through the sample replay method is as follows:

[0088] Filter the clustering results. By calculating the silhouette coefficient of each clustering heap, high-quality clustering clusters are selected according to the silhouette coefficient, and clustering clusters with a low number of samples are filtered out. The finally retained clustering clusters are manually mapped with pseudo-labels as training samples for the new categories of incremental learning, and the open-world object detector is incrementally learned through the sample replay method.

[0089] Figure 5 This is the structural diagram of the semi-supervised contrast clustering module of the present invention;

[0090] The semi-supervised contrast clustering model includes a strong data and weak data augmentation module, a ResNet34 feature extraction module, an instance-level contrast clustering module, and a clustering-level contrast clustering module;

[0091] The strong data and weak data augmentation module is used to obtain a set of strongly augmented and two sets of weakly augmented images based on the unknown objects output by the open-world object detector model and the predicted known objects;

[0092] The ResNet34 feature extraction module is used to extract features from the images augmented by the strong data and weak data augmentation module;

[0093] The instance-level contrast clustering module is used to output instance-level features based on the features extracted by the ResNet34 feature extraction module; the structure of the instance-level clustering module includes a two-layer MLP;

[0094] The clustering-level contrast clustering module is used to output clustering results based on the instance-level features output by the instance-level contrast clustering module, and then calculate the instance-level and clustering-level similarities to maximize the similarity of positive sample pairs. The structure of the clustering-level clustering module is: a two-layer MLP and a Softmax function;

[0095] Embodiment 1:

[0096] This embodiment is built with 1 GTX 3090 graphics card server and 1 Nvidia Jetson Xavier mobile lightweight computer as the test running environment for the method of the present invention. All experiments are conducted on the server for model training and the model is tested and deployed on the Nvidia Jetson Xavier. The specific hardware configuration information is shown in Table 1 and Table 2.

[0097] Table 1 Server Hardware Configuration

[0098]

[0099]

[0100] Table 2 Nvidia Jetson Xavier Hardware Configuration

[0101] Configuration Specification CPU 6-core NVIDIA Carmel ARM v8.2 64-bit CPU GPU 384-core NVIDIA Volta GPU with 48 Tensor cores Memory 16GB Hard Disk 500GB

[0102] This embodiment uses Pycharm as the development environment for the method of the present invention and Python as the programming language to complete the method design and development. The software environment included in this embodiment for the method to run includes: operating system Ubuntu 16.04.01, Python 3.8. The specific software environment is shown in Table 3.

[0103] Table 3 Software Environment

[0104] Software Environment Version Operating System Ubuntu 16.04.01 Python Version 3.8 Development Environment Pycharm Language Python

[0105] This embodiment uses the Pascal VOC and COCO datasets as the experimental datasets. Among them, we set the 20 categories included in Pascal VOC as the known class training set, and we screen the COCO dataset for testing. We select 504 images with 1655 unknown object annotations from the COCO dataset as the COCO - OOD and 897 images containing 2658 known objects and 2533 unknown objects simultaneously from the COCO dataset as the COCO - Mixed dataset for testing.

[0106] An open - world object detection method based on contrastive clustering and multivariate Gaussian distribution, the detailed steps are as follows:

[0107] (1) Collect image data (without ground - truth labels) in the open - world scene through an existing dataset (containing ground - truth labels) or an image acquisition device.

[0108] (2) Construct an open - world object detector model and train the open - world object detector model; First - stage training: Pass the original image data through the first feature extraction network of the object localization module, ResNet50; for feature extraction;

[0109] (3) Generate and predict candidate boxes for object features through the Region Proposal Network (RPN) with predicted centrality C, where the prediction of foreground scores and the widths and heights of predicted boxes are replaced by the centrality C and the distances from the anchor point (x, y) to the four sides of the predicted box.

[0110] The definition of centrality is as follows:

[0111] l * = x – x0 (1)

[0112] r * = x1 – x (2)

[0113] t * = y – y0 (3)

[0114] b * = y1 – y (4)

[0115]

[0116] Where: Define the ground truth box as B = (x0, y0, x1, y1), and x0 < x < x1, y0 < y < y1, l * , r * , t * , b * represent the distances from the anchor point to the left, right, top, and bottom of the ground truth box, and the ground truth centrality of each anchor point is calculated by formula (5).

[0117] Input the features output by ResNet50 of the object localization module and the predicted candidate boxes into the ROI head, extract features from each layer of the feature pyramid through RoIAlign, and then perform fine-grained regression of the candidate boxes and prediction of the intersection over union (IoU) score I; where the ground truth of the IoU score is the IoU of each predicted box and the corresponding ground truth box.

[0118] (5) Obtain the final object score S for each candidate box and filter the candidate boxes. The calculation of the object score S is as follows:

[0119]

[0120] (6) In the second-stage training, freeze the network parameters of the object localization module, input the image into the object localization module and ResNet50 to obtain object candidate boxes and the feature representation of the image;

[0121] (7) Fit the class-conditional multivariate Gaussian distribution N(μ, Σ), where the mean and variance are calculated by the following formulas:

[0122]

[0123] Where: N k represents the total number of objects in the k-th category, and h(x i ) represents the features extracted from each candidate box.

[0124] (8) Perform classification prediction, calculate the classification loss, and perform gradient backpropagation;

[0125] (9) In the third-stage training, continue to freeze the network parameters of the object localization module, start generating the pseudo-feature v, and send the pseudo-feature v into the classification network to obtain the classification output. Calculate the energy score E(x; θ) of the pseudo-feature v and the classification output of the normal input image, and use the energy score for contrastive learning to make the energy score of the pseudo-feature v have a lower value. The pseudo-feature generation and energy function formulas are as follows:

[0126]

[0127] Where: represents the sampling point of the k-th category, and ∈ is a very small value. f cls (x; θ) represents the output result of the classification network of the sample, and w k is a learnable parameter. In the experiment, we sample 10,000 samples from the multivariate Gaussian distribution of each category and select the sample with the lowest probability to replace the selection of ∈.

[0128] (10) Select a threshold γ. Those with an energy score higher than the threshold are known categories, and those lower than the threshold are unknown objects; in the experiment, the value at which 95% of the normal input samples in the training set can be correctly classified into known categories is selected as the threshold.

[0129] (11) We conducted detection tests on known and unknown objects on the Pascal VOC and COCO datasets, as shown in the following table.

[0130] Table 4

[0131]

[0132]

[0133] As can be seen from Table 4, while ensuring the accuracy of known category detection, this method shows good results in scenarios containing only unknown objects and scenarios containing both known and unknown objects. It can be seen from the indicators that our method can detect more unknown boxes. Through the multi-stage training method, our method maintains the accuracy of known object recognition and is superior to other methods in the localization of unknown objects, with an average precision superior to other methods.

[0134] Perform contrastive clustering learning on the objects not recognized by the open-world object detector model based on a semi-supervised contrastive clustering model to obtain contrastive clustering results;

[0135] S31: Use the detected unknown objects (without ground truth labels) and the predicted known objects (with ground truth labels) as data inputs, and perform contrastive clustering in a semi-supervised manner. Define the augmented image pairs from the same instance as positive sample pairs, and define other image pairs as negative sample pairs. (2) We perform data augmentation on the input data. Given a batch of samples of size N, after performing data augmentation on each instance of {X1,…,X N}, we can obtain 3N samples Among them, the samples with strong data augmentation are The remaining two groups are samples with weak data augmentation The augmented image pairs of the same instance will be regarded as positive pairs. For example and Secondly, if two samples X i and X j , have the same label, then and will both be regarded as positive sample pairs. In addition, the pairing between the two augmentations and will also be regarded as a positive pairing. If or does not meet the above conditions, it is regarded as a negative pair. For the unknown object X i , only the augmented image pair will be regarded as a positive image pair, and other image pairs are negative image pairs. Among them, we group all the augmented samples in pairs and calculate the loss of each group. For example, the instance-level loss between the strongly data-augmented sample and the weakly data-augmented sample is defined as L ins(a,b) .

[0136] S32: For the features F extracted by the backbone network ResNet34 from the images after feature augmentation

[0137] S33: The calculation process of the overall loss of the instance-level contrastive clustering module is as follows:

[0138] Use the first double-layer MLP to obtain the feature representation I of the object i , and at the same time use the cosine distance as the similarity. Taking the strongly data-augmented sample and the weakly data-augmented sample as an example, the specific formula is as follows:

[0139]

[0140] Where: t1,t2∈{a,b},i,j∈[1,N], the contrast loss of the sample combination is defined as follows:

[0141]

[0142] {k +} is the same as A set of pairs of instances; {k -} is the same as The set of negative pairs of instances, T I is the instance-level temperature parameter used to control softness. I Set to 1.5, the instance-level contrast loss is calculated as follows:

[0143]

[0144] The final instance-level overall loss is composed of the strongly enhanced samples and weakly enhanced samples With weakly enhanced samples and weakly enhanced samples The specific formula is as follows.

[0145] L ins =L ins(a,b) +L ins(b,c) (14)

[0146] S34: The calculation process of the overall loss of the cluster level comparison cluster module is as follows:

[0147] The structure of cluster-level contrast clustering is similar to that of instance-level contrast clustering. Both known objects and unknown objects are used for cluster-level contrast clustering. In actual use, known objects are only used to enhance the clustering effect. After clustering, the actual use will ignore the known object clustering stack. Another two-layer MLP is used to project the features output by ResNet34 into M-dimensional space, where M is the number of clusters, to enhance the samples with strong data. Samples with weak data augmentation For example, given an enhanced input Output an N×M matrix, To enhance the representation of the mth cluster of sample a, C a and C b There are 2M columns in total, ignoring and Oneself, and This pair is considered a positive sample pair, and all remaining sample pairs are defined as negative sample pairs. The cosine distance is used to calculate the two clusters. The pairwise similarity, where m, n ∈ [1, M]. The following clustering loss and contrastive loss reduce the distance between two augmented representations in the same cluster. The calculation formula is as follows:

[0148]

[0149] where t1, t2 ∈ {a, b}, T C is the temperature coefficient, is the clustering set that forms negative sample pairs with The clustering-level loss function is defined as follows:

[0150]

[0151] where, is the entropy of the cluster assignment probability under each feature augmentation representation in the batch. By penalizing a high level of clustering assignment uncertainty, the model is encouraged to avoid assigning most instances to one cluster; The final clustering-level overall loss is composed of strongly augmented samples and weakly augmented samples Weakly augmented samples and weakly augmented samples and strongly augmented samples and weakly augmented samples together. The specific formula is as follows:

[0152] L clu = L clu(a,b) + L clu(a,c) + L clu(b,c) (18). The final loss of this module is jointly obtained by the instance-level contrastive clustering module and the clustering-level contrastive clustering module. The formula is as follows:

[0153] L = L clu + L ins (19)

[0154] S35: After clustering, M clusters will be obtained from all known and unknown categories. Assuming there are K types of known objects, then there will be K corresponding clusters. We use the ground truth of the known objects to filter out the clusters that match each known object. P i = argmax(C i ) is the probability of the sample X i belonging to the cluster, and the number of known objects is M. Therefore, we can calculate the confusion matrix

[0155] on the probability of the known objects belonging to the cluster and the ground truth label The calculation method of the cost matrix is as follows:

[0156]

[0157] By minimizing the cost matrix, we can find the best one-to-one matching between M clusters and M labels.

[0158]

[0159] P is the set of all possible one-to-one matchings. Among the M of the best matchings, if the i-th cluster is matched with the j-th known object label, then this cluster will be ignored when generating pseudo-labels.

[0160] To test the ability to discover new categories in unknown objects, we conducted relevant experiments on the VOC2012 and COCO datasets. The COCO dataset includes a total of 80 categories. Referring to the settings of the open-world object detection T1 task, we used 60 categories outside the labeled categories in the VOC dataset as unknown categories for clustering.

[0161] Table 5

[0162] Clustering Method NMI ACC Purity K-means 18.2 12.4 17.0 FINCH 4.8 12.2 9.7 DTC+ 21.8 20.1 28.5 OSODD 19.1 14.5 17.9 OWDet 37.9 21.3 35.3 This Method 39.2 30.4 37.6

[0163] As can be seen from Table 5, our semi-supervised contrastive clustering method far outperforms several other baseline methods in all three tasks. In contrast, the semi-supervised clustering method is stronger than the unsupervised clustering in terms of clustering effect. Moreover, we use strong / weak data augmentation, and by contrastive learning of strong augmented samples and weak augmented samples, the robustness of clustering is enhanced.

[0164] Based on the contrastive clustering results output by the semi-supervised contrastive clustering model in an incremental learning manner, select the cluster with the best clustering, assign pseudo-labels, and return to the open-world object detector model through the way of sample replay; the process is as follows:

[0165] S41: Screen out the clusters with samples less than a specific threshold r. The relevant formula for the silhouette coefficient is as follows:

[0166]

[0167] Assume that a sample X belongs to the m-th cluster, then the silhouette coefficient of X is s(X), where a(X i ) is the average distance of X i from other samples belonging to the same cluster C m ; b(X i ) is the minimum distance of X from other clusters;

[0168]

[0169] where dist(X i , X j ) is the distance between X i and X jThe square root of the distance between them. We define the silhouette coefficient of a cluster as the average silhouette coefficient of all samples belonging to that cluster:

[0170]

[0171] S42: Sort all clusters according to the silhouette coefficient, and manually map the pseudo-labels of the top k clusters to real labels.

[0172] S43: Combine the new k categories into a new data set and train an open-world object detector.

[0173] S44: Select a partial sample set of known categories to form a small sample set, and use this small sample set for sample replay to avoid the catastrophic forgetting problem of the open-world object detector for previously learned known categories.

[0174] In the experiment, train for a fixed number of iterations and select the model after the iteration is completed. In the specific application stage, an open-world object detector model with the best performance on the validation set among all iterations can be selected;

[0175] Figure 6 This is the effect diagram of detecting known and unknown objects by this method; this is the visualization diagram of some results of the present invention on the COCO data set, and it can be seen that our method can effectively distinguish known and unknown objects in the image.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open-world object detection method based on contrastive clustering and multivariate Gaussian distribution, characterized in that: Including the following steps: Obtain the image to be detected; Construct an open-world object detector model for detecting known and unknown objects in the image; Train the open-world object detector model; Perform contrastive clustering learning on the objects not recognized by the open-world object detector model based on the semi-supervised contrastive clustering model to obtain the contrastive clustering result; Based on the incremental learning method, select the best-clustered heap from the contrastive clustering results output by the semi-supervised contrastive clustering model, assign pseudo-labels, and return it through the sample replay method, and return to train the open-world object detector model; When the training of the open-world object detector model reaches a fixed number of iterations, obtain the trained open-world object detector model; Input the image to be detected into the trained open-world object detector model to realize the category recognition of the image to be detected.

2. The open-world object detection method based on contrast clustering and multivariate Gaussian distribution according to claim 1, characterized in that: The open-world object detector model includes an object localization module for detecting the position of the target in the image; The object localization module includes a first feature extraction network for extracting the features of the target in the image; Object candidate network module: used to predict the centrality of the target in the image and generate candidate boxes; Two-stage regression network: Based on the features extracted by the first feature extraction network and the predicted candidate boxes output by the object candidate network module, extract features from each layer of the feature pyramid, and then perform refined regression of the candidate boxes and prediction of the intersection over union score I.

3. An open-world object detection method based on contrast clustering and multivariate Gaussian distribution according to claim 1, characterized in that: The open-world object detector model includes a classification module for classifying the target in the image based on the target position output by the object localization module; The classification module includes: a second feature extraction network for extracting the features of the target in the image; RoI Align: Based on the features extracted by the second feature extraction network and the box output by the object localization module, sample the features output by the second feature extraction network according to the output box; Pseudo-feature generation module: used to generate pseudo-features based on the sampled features output by RoI Align; Classification network: used to match the target in the image with the known categories based on the sampled features output by RoI Align and the pseudo-features output by the pseudo-feature generation module.

4. An open-world object detection method based on contrastive clustering and multivariate Gaussian distribution according to claim 1, characterized in that: The process of training the open-world object detector model is as follows: The first stage: Train the object localization module; The second stage: Freeze the network parameters of the object localization module, fit with a multivariate Gaussian distribution, and start training the classification module: The third stage: Continue to train the classification module, generate pseudo-features according to the fitted Gaussian distribution, the pseudo-features of each category will be input into the classification network, obtain the classification results and calculate the energy values, and through contrastive learning with the normally extracted features, let the classification network learn the boundaries of each category; The fourth stage: Calculate the threshold to distinguish known and unknown objects.

5. The open-world object detection method based on contrast clustering and multivariate Gaussian distribution according to claim 1, characterized in that: The semi-supervised contrastive clustering model includes a strong data and weak data augmentation module, a ResNet34 feature extraction module, an instance-level contrastive clustering module, and a clustering-level contrastive clustering module; The strong data and weak data augmentation module is used to obtain a set of strongly augmented and two sets of weakly augmented images based on the unknown objects output by the open-world object detector model and the predicted known objects; The ResNet34 feature extraction module is used to extract features from the images augmented by the strong data and weak data augmentation module; The instance-level contrast clustering module is used to output instance-level features based on the features extracted by the ResNet34 feature extraction module; The clustering-level contrast clustering module is used to output a clustering result based on the instance-level features output by the instance-level contrast clustering module, and then calculate the instance-level and clustering-level similarities to maximize the similarity of positive sample pairs.

6. The open-world object detection method based on contrast clustering and multivariate Gaussian distribution according to claim 5, wherein The calculation process of the overall loss of the instance-level contrast clustering module is as follows: Obtain the feature representation I of the object using the first double-layer MLP i , while using the cosine distance as the similarity, taking the strongly data-augmented samples and the weakly data-augmented samples as an example, the specific formula is as follows: where: t1, t2 ∈ {a, b}, i, j ∈ [1, N], and the contrast loss of the sample combination is defined as follows: {k + } is the same as A set of pairs of instances; {k - } is the same as The set of negative pairs of instances, T I is the instance-level temperature parameter used to control softness. I Set to 1.5, the instance-level contrast loss is calculated as follows: The loss of the final instance-level ensemble consists of strongly augmented samples and weakly augmented samples along with weakly augmented samples and weakly augmented samples and is formulated as follows. L ins = L ins(a,b) + L ins(b,c) (14).

7. An open-world object detection method based on contrast clustering and multivariate Gaussian distribution according to claim 5, characterized in that: The calculation process of the overall loss of the clustering-level contrast clustering module is as follows: Another double - layer MLP is adopted to project the features extracted by ResNet34 into an M - dimensional space, where M is the number of clustering clusters, for strongly data - augmented samples and weakly data - augmented samples For example, given an augmented input output an N×M matrix, which is the representation of the m - th cluster of the augmented sample a, C a and C b There are a total of 2M columns, ignoring itself, and This pair is regarded as a positive sample pair, and all the remaining sample pairs are defined as negative sample pairs. The cosine distance is used to calculate the pairwise similarity of the two clusters where m,n∈[1,M]. The following clustering loss and contrastive loss reduce the distance between the two augmented representations of the same cluster, and the calculation formula is as follows: where \(t_1, t_2\in\{a, b\}\), \(T\) C is the temperature coefficient, is related to the clustering set that forms negative sample pairs, and the clustering-level loss function is defined as follows: Among them, is the entropy of the cluster assignment probability under each feature augmentation representation in the batch. By penalizing a high level of clustering assignment uncertainty, the model is encouraged to avoid assigning most instances to one cluster; the final loss of the overall clustering level is composed of strongly augmented samples and weakly augmented samples Weakly augmented samples and weakly augmented samples together with strongly augmented samples and weakly augmented samples and is specifically formulated as follows: L clu = L clu(a,b) + L clu(a,c) + L clu(b,c) (18); The final loss of this module is jointly obtained by the instance-level contrast clustering module and the clustering-level contrast clustering module, and the formula is as follows: L = L clu + L ins (19).

Images