An open set semi-supervised object detection method based on contrastive learning

Abstract

The present application belongs to the field of computer vision, and discloses an open set semi-supervised object detection method based on class contrast learning. The present application proposes a solution for the instance-level approximate out-of-distribution data problem, combines the clustering idea and the class contrast learning idea, proposes an approximate out-of-distribution data filtering strategy, abstractly divides a dynamic boundary for each predicted instance, and prevents the error semantic expansion caused by the approximate out-of-distribution data instance. Through the effective integration of the dynamic threshold strategy and the class contrast learning, the over-strict out-of-distribution anomaly filtering in the early training stage can be prevented, the model can be prevented from misjudging too many normal instances as out-of-distribution instances, and the generalization performance of the model can be affected. In addition, the generalization ability of the model can be improved by using the normal instances and a small amount of approximate out-of-distribution data instances in the unlabeled data, and the label allocation can be strictly controlled in the later training stage, so that too many approximate out-of-distribution instances are prevented from being allocated normal pseudo-labels, and the model performance is prevented from rapidly declining due to too large data variance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target detection of computer vision, and relates to an open set semi-supervised target detection method with inconsistent data label space. BACKGROUND

[0002] With the rapid development of deep learning technology, many researchers use this advanced technology based on data driving to achieve performance comparable to or even surpassing human performance in many tasks in the field of computer vision. However, the outstanding performance of deep learning largely depends on high-quality data annotation. In order to reduce the time-consuming and labor-intensive annotation burden, deep semi-supervised learning has gradually become the research focus of many researchers due to its need for only a small amount of labeled data. Among them, the label data is also called In Distribution (ID) data. In recent years, deep semi-supervised learning has been widely studied in image classification tasks and target detection tasks. In the classification task, such as Dash (Xu Y, Shang L, Ye J, et al. Dash: Semi-supervised learning with dynamic thresholding [C] / / International conference on machine learning. PMLR, 2021: 11525-11536.) first considers using dynamic thresholding to filter the pseudo label more reasonably when screening the pseudo label, and in the target detection task, such as Unbiased teacher (Liu YC, Ma C Y, He Z, et al. Unbiased teacher for semi-supervised object detection [J]. arXiv preprint arXiv: 2102.09480, 2021.) uses teacher model and student model to learn from each other, which provides a basic framework for the development of the subsequent open set semi-supervised target detection field.

[0003] Although the development of existing deep semi-supervised learning significantly reduces the dependence of the algorithm on ID data, most of these studies are based on a core assumption that ID samples and unlabeled samples follow the same data distribution, that is, the independent and identically distributed assumption. However, this basic assumption does not always hold in the real world open environment. Specifically, there are a large number of OUT OF DISTRIBUTION (OOD) data in the unlabeled data set, which often leads to false semantic expansion of existing semi-supervised learning algorithms, thereby seriously reducing the model performance.

[0004] To this end, many researchers consider solving the OOD problem in the framework of deep semi-supervised learning to screen pseudo labels, thereby reducing the expansion of false semantics. Among them, in the semi-supervised classification task, OpenMatch (Saito K, Kim D, Saenko K. Openmatch: Open-set semi-supervised learning with open-set consistency regularization[J]. Advances in Neural Information Processing Systems, 2021, 34: 25956-25967.) trains two classifiers using labeled data, one is a traditional classifier used to determine the class to which the data belongs, and the other is a One vs All classifier (Saito K, Saenko K. Ovanet: One-vs-all network for universal domain adaptation[C] / / Proceedings of the ieee / cvf international conference on computer vision. 2021: 9000-9009.), which aims to distinguish one class from other classes, and through the cooperation of the two classifiers, pseudo labels are assigned. CCSSL (Yang F, Wu K, Zhang S, et al. Class-aware contrastive semi-supervised learning[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 14421-14430.) introduces a class-aware contrastive learning method to improve performance in open-set settings. UnMixMatch (Roy S, Etemad A. Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(13): 14847-14856.) differs from the method of filtering OOD data and then assigning pseudo labels, but integrates self-supervised learning with semi-supervised learning, making full use of unlabeled data to enhance the model's ability to extract features. These studies have expanded the ideas for object detection tasks.

[0005] There are also related researches on semi-supervised object detection tasks to solve the OOD problem. For example, OSSOD (Liu Y C, Ma C Y, Dai X, et al. Open-set semi-supervised object detection [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 143-159.) adds a plug-and-play Dino pre-training module to the basic framework of semi-supervised object detection, and considers using background class instances as OOD classes to fine-tune the Dino module to assist in filtering OOD instances. Subsequently, OOSSOD (Wang Z, Xiao L, Xiang L, et al. Online Open-set Semi-supervised Object Detection with Dual Competing Head [J]. arXiv preprint arXiv:2305.13802, 2023.) proposes an end-to-end online framework, which draws on the idea of noisy labels and trains a pair of dual competing classification heads to filter OOD instances.

[0006] In the present application, we found that in the OOD data of the target detection task, in addition to the common OOD data with large gap from the ID data, there is also a kind of distribution outside data with certain similarity with the labeled data, namely Near OOD. In order to distinguish Near OOD data, we call the common OOD data Far OOD. Although Near OOD data is not included in the ID data set, they have certain similarity with ID data, so it is difficult for visual processing model to identify and detect them, which will lead to the performance reduction of the model due to the too large data variance. Yang et al. (Yang J, Zhou K, Liu Z. Full-spectrum out-of-distribution detection. arXiv 2022[J]. arXiv preprint arXiv:2204.05306.) once explored the Near OOD problem. However, the existing open set semi-supervised target detection method has not considered the Near OOD problem, so it will produce false semantic generalization because of ignoring this part of data, and further lead to the decline of model detection performance. Therefore, the present application proposes to deeply mine the unique properties of Near OOD instances in the open set semi-supervised target detection task, and adaptively affect the iterative process of the target detection model. In the early stage of model training iteration, the similarity between Near OOD instances and conventional data is used to enhance the learning ability of the model to the characteristics. As the iteration goes deeper, because the model gradually establishes strong learning ability, in the later stage of iteration, it turns to more explicit OOD detection to assist the model to effectively identify and exclude Near OOD, so as to learn more explicit sample characteristics, so as to avoid the potential interference of Near OOD instances on the prediction accuracy of the model. Therefore, by solving the Near OOD problem, the present application further improves the accuracy of open set semi-supervised target detection. SUMMARY

[0007] To solve the problem of the existence of Near OOD data interfering with the performance of open set semi-supervised target detection, the application proposes an improved OSSOD framework based on class contrast learning, which can effectively learn accurate features by more effectively utilizing ID data to filter Near OOD data, thereby improving the detection accuracy of the model. First, this paper adopts a class contrast learning strategy, which can narrow the similarity distance between instances of the same class and widen the similarity distance between instances of different classes, so as to make the class boundary of ID data more clear, thereby significantly improving the utilization efficiency of labeled data. Through this method, the model can more accurately distinguish different class ID instances and enhance the adaptability to complex data distribution. In addition, in order to prevent the negative impact of false semantic expansion on the performance of the model, this paper draws on the clustering idea and uses the similarity distance of class contrast learning for driving. Specifically, this research stores a small amount of diverse instances for each ID class to simulate the cluster boundary of each class. In order to reduce the amount of calculation and consider the particularity of the open set, we do not use the traditional clustering method to find the center point. We use the nearest ID instance to the predicted instance as the anchor, and the similarity distance between the anchor and the most diverse ID instance in the same class as the threshold. In this way, we abstractly realize a dynamic threshold for each class and even each instance, thereby more effectively improving the out-of-distribution detection performance.

[0008] Finally, this paper proposes an ensemble learning strategy for dynamically regulating the OOD filtering mechanism. This strategy can prevent excessive strict OOD filtering from causing the model to misjudge too many ID instances as OOD instances in the early stage of training, thereby affecting the generalization performance of the model, and can strictly control the label assignment in the later stage of training to prevent too many Near OOD instances from being assigned ID pseudo-labels, which can cause the model performance to rapidly decline due to excessive data variance.

[0009] The technical scheme of the application:

[0010] An open set semi-supervised target detection method based on class contrast learning, the steps are as follows:

[0011] Step 1: According to the label data of each class label instance level data, adaptively allocate different size feature memory buffer;

[0012] According to the number of instance level data of each class in the labeled data, the linear interpolation and the scale factor are mapped to map the instance level data of each class to the feature memory buffer, and the adaptive allocation of different size class feature memory buffer is performed for each class instance level data. The class feature memory buffer mainly stores the annotated instance features in a first-in-first-out manner;

[0013] Step 2: The labeled data is input into the Faster-RCNN model for supervised learning training, and the class contrast learning is used to narrow the intra-class distance and widen the inter-class distance, and the intra-class memory buffer is updated after iteration;

[0014] The labeled data is weakly augmented, and the augmented instance-level data is aligned with the label. The aligned data is input into the Faster-RCNN model for supervised learning training. After training, the most similar instance-level data is updated to the feature memory buffer of the corresponding class according to the similarity distance The least similar instance-level data of the same class is updated to the feature memory buffer of the corresponding class. The supervision loss is: (1) the RPN module classification loss, RPN module regression loss, ROI module classification loss, and ROI module regression loss for target detection, which is used to improve the target detection accuracy; (2) the class contrast learning loss, which narrows the intra-class distance of the same class label data and widens the inter-class distance of different class label data to improve the feature extraction ability of the Faster-RCNN model and improve the classification accuracy of the Faster-RCNN model. The weights of the trained Faster-RCNN model are assigned to the teacher model and the student model. The teacher model and the student model have the same structure as the trained Faster-RCNN model, so they have the same weights and similar predictions. The class contrast learning loss is as follows:

[0015]

[0016] Where i, j, k represent the index subscript; z i represents the index i; represents that k, i belong to the same class; represents the indicator function, which is 0 when j = i, otherwise 1; N represents the number of instance features of class c; L c,i represents the contrast loss of the instance feature with index subscript i of the fixed anchor of class c; L c represents the contrast loss of class c; P(i) represents the number of other instance features with the same class as the fixed anchor with index subscript i; τ is the temperature coefficient;

[0017] Step 3: Repeat step 2 for 2K times, and copy the weights to the teacher model and the student model after iteration is completed, both of which share the same weights;

[0018] Step 4: The unlabeled data is weakly augmented and input into the teacher model for prediction to obtain the to-be-screened pseudo label;

[0019] Step 5: The generated to-be-screened pseudo label is adaptively screened. Before a certain number of iterations, either of the two screening mechanisms determines that the pseudo label is true, i.e. true. After a certain number of iterations, both screening mechanisms need to determine that the pseudo label is true, i.e. true.

[0020] The to-be-screened pseudo label is adaptively screened, and the screening mechanism is dynamically regulated by two filtering methods through voting in ensemble learning and iteration times. The first filtering method is: comparing the predicted value of the instance feature corresponding to the to-be-screened pseudo label with a dynamic threshold, and discarding the instance feature and the pseudo label lower than the dynamic threshold; wherein the dynamic threshold is an initial value of 0.3, and is increased to 0.8 within a certain round according to a linear function. The second filtering method is: finding the most similar instance feature to the current predicted instance in the feature memory buffer area corresponding to the to-be-screened pseudo label, denoted as anchor point z anchor ; then, the similarity distance between the anchor point and the least similar instance feature in the feature memory buffer area corresponding to the category is taken as the threshold SIM threshold ; if the similarity distance SIM {pred,anchor} between the instance feature corresponding to the current to-be-screened pseudo label and the anchor point is greater than the threshold, the current predicted instance pseudo label is retained, otherwise it is discarded; wherein z anchor , SIM threshold and SIM {pred,anchor} are obtained by the following formulas respectively:

[0021]

[0022]

[0023] SIM {pred,anchor} =z pred ·z anchor

[0024] Wherein z represents an instance feature, z pred represents the instance feature corresponding to the to-be-screened pseudo label, z l represents the instance feature in the feature memory buffer area of the same category as the to-be-screened pseudo label, Minimize() represents the instance feature z l with the minimum value in the multiplication of multiple points as the anchor point z anchor , z l1 , z l2 represent different instance features in the same category feature memory buffer area, Max() represents the maximum value in the multiplication of multiple points as the threshold SIM threshold , SIM {pred,anchor} represents the multiplication of the instance feature corresponding to the to-be-screened pseudo label and the corresponding anchor point instance feature.

[0025] The two filtering methods are dynamically regulated according to the voting in ensemble learning, and any one of the two filtering methods determines the pseudo label as true before the iteration times of N times the number of label data, and both are retained; if the iteration times are exceeded, both methods need to determine that the pseudo label is true, and the pseudo label is retained.

[0026] Step 6: After the pseudo-labels are assigned to the unlabeled data corresponding to the instance-level data, the labeled data and the unlabeled data are sent into the student model for training, wherein the unlabeled data is only trained by the supervised loss, the labeled data is additionally trained by the class contrast learning loss, and the most dissimilar instance features in the labeled data are updated to the feature memory buffer; finally, the student model is updated to the teacher model according to the exponential moving average method, wherein a = 0.1;

[0027] Step 7: Repeat steps 4 to 6 to reach the specified training steps, and apply the final teacher model to the test environment and perform prediction.

[0028] Advantages of the present application: In the scene of semi-supervised target detection, the present application uses a basic dynamic threshold and a class contrast learning driven method to screen the approximate out-of-distribution samples, and realizes an end-to-end trainable open set semi-supervised target detection framework. For the first time, a solution is proposed for the instance-level approximate out-of-distribution data problem, an innovative approximate out-of-distribution data filtering strategy is proposed by combining the clustering idea and the class contrast learning idea, and a dynamic boundary is abstractly divided for each predicted instance, thereby effectively preventing the error semantic expansion caused by the approximate out-of-distribution data instances. Through the effective integration of the basic dynamic threshold strategy and the class contrast learning, it can prevent the over-strict out-of-distribution anomaly filtering from causing the model to misjudge too many normal instances as out-of-distribution instances in the early training stage, thereby affecting the generalization performance of the model, and it can also use the normal instances and a small amount of approximate out-of-distribution data instances in the unlabeled data to improve the generalization ability of the model, and it can strictly control the label allocation in the later training stage to prevent too many approximate out-of-distribution instances from being assigned normal pseudo-labels, which causes the model performance to rapidly decline due to the excessive variance of the data. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A detailed training framework structure diagram of the open set semi-supervised target detection based on class contrast learning provided by the present application;

[0030] Figure 2 A feature memory buffer division schematic diagram provided by the present application;

[0031] Figure 3 A pseudo-label filtering schematic diagram based on similarity distance provided by the present application. DETAILED DESCRIPTION

[0032] The specific embodiments of the present application are further described below in combination with the drawings and technical solutions

[0033] The application provides an open set semi-supervised object detection method based on contrastive learning, and the specific implementation process is as follows: first, a small feature memory buffer is opened in the memory, and the total memory size is N times the number of marked data categories, and N is generally 16; in order to allocate different sizes of feature memory buffers to each category, after counting the number of instance features of each category in the marked data, linear interpolation and a scale factor are used to calculate the size of the feature memory buffer of each category, wherein the size of the feature memory buffer is an even number, and the access of instance features is carried out in the way of queue first-in-first-out in the data structure; as shown in Figure 2 . After allocating the feature memory buffer of each category, two-stage training is started, as shown in Figure 1 . First, the marked data is sent into the Faster-RCNN model for a certain round of supervised loss training after random data weak enhancement, and when performing contrastive learning, the instance features with the same category but the least similarity in a batch of labeled data are stored in the feature memory buffer of the corresponding category. When the first stage of training is completed, the weight of the trained Faster-RCNN model is assigned to the teacher model and the student model, and the teacher model and the student model have the same structure as the trained Faster-RCNN model, so that they have the same weight, similar prediction, and start the second stage of training. In the second stage of training, first, the unmarked data is sent into the teacher model for prediction after random data weak enhancement, and the instance features in the unmarked data are assigned with their predicted labels, also known as pseudo labels. After obtaining the pseudo labels, they need to be filtered. The filtering mechanism is that before a certain number of iterations, two filtering methods are used to determine that the pseudo label predicted by the instance feature is wrong, and the pseudo label will be discarded. After a certain number of iterations, any one of the filtering methods determines that the pseudo label predicted by the instance feature is wrong, and the pseudo label will be discarded. The two screening methods are as follows: (1) dynamic threshold filtering method, an initial threshold of 0.3 is set, and a final threshold of 0.8 is set, and the linear function is used to calculate the dynamic threshold value, wherein n is the total number of iterations, and when the maximum prediction value of the instance feature is less than the dynamic threshold value, the pseudo label corresponding to the instance feature is determined to be wrong; (2) pseudo label filtering method based on similarity distance, find the most similar labeled instance feature to the current instance feature in the feature memory buffer of the category to which the pseudo label of the instance feature belongs, and mark it as anchor z anchor . Then, find the instance feature in the same category feature memory buffer that is the least similar to the anchor, and calculate the similarity distance between the anchor and the instance feature as the threshold SIM threshold . If the similarity distance SIM {pred,anchor} between the instance feature to be determined and the anchor is less than the threshold, the pseudo label corresponding to the instance feature to be screened is determined to be wrong, as shown in Figure 3 .

[0034] After filtering the pseudo labels, the pseudo labels are assigned to the unlabeled data, and the data assigned with the pseudo labels and the labeled data are sent into the student model for supervised loss learning after random data strong enhancement, wherein the labeled data is subjected to class contrast learning, and the same class and the least similar instance features are also stored into the corresponding feature memory buffer. After one training is completed, the weight of the student model is updated to the teacher model by exponential moving average. After repeated training for a certain number of rounds, the weight of the teacher model is fixed, and is applied to the test environment.

[0035] Overall, the present application solves the problem of Near OOD instance recognition mainly through class contrast learning and voting mechanism in ensemble learning by using the open set semi-supervised target detection model with dynamic regulation and class contrast learning. Specifically, the combination of dynamic threshold and voting mechanism not only reduces the complexity of setting hyperparameters, but also effectively improves the robustness of the model through the phased regulation mechanism, avoids the performance degradation of the model due to error accumulation, and can learn from Near OOD to a certain extent.

Claims

1. An open-set semi-supervised image object detection method based on contrastive learning, characterized in that, The steps are as follows: Step 1: According to the label data of each category, the label instance level data is adaptively allocated to different size feature memory buffer; Step 2: The labeled data is sent into the Faster-RCNN model for supervised learning training, while the class contrast learning is used to pull the intra-class distance and push the inter-class distance, and the memory buffer of each category is updated after iteration; Step 3: Repeat step 2 for 2K times, and copy the weight as the teacher model and student model after iteration, both of which share the same weight; Step 4: The unlabeled data is sent into the teacher model for prediction after data weak enhancement, and the pseudo label to be screened is obtained; Step 5: The generated pseudo label to be screened is adaptively screened, and before a certain number of iterations, any one of the two screening mechanisms determines that the pseudo label is true, that is, it is true; After a certain number of iterations, both screening mechanisms need to determine that the pseudo label is true, and it is true; The to-be-screened pseudo label is adaptively screened, and a screening mechanism is two filtering methods for dynamic regulation through voting in ensemble learning and iteration times. The first filtering method is: comparing the predicted value of the instance feature corresponding to the to-be-screened pseudo label with a dynamic threshold, and discarding the instance feature and the pseudo label lower than the dynamic threshold; wherein the dynamic threshold is an initial value of 0.3, and is increased to 0.8 within a certain round according to a linear function. The second filtering method is: finding the most similar instance feature to the current predicted instance in the feature memory buffer area corresponding to the to-be-screened pseudo label, which is recorded as an anchor point ; then, the similarity distance between the anchor point and the least similar instance feature in the feature memory buffer area corresponding to the category is taken as a threshold ; if the similarity distance between the instance feature corresponding to the current to-be-screened pseudo label and the anchor point is greater than the threshold, the current predicted instance pseudo label is retained, otherwise it is discarded . Step 6: The pseudo label is assigned to the instance level data corresponding to the unlabeled data, and then sent into the student model together with the labeled data for training, wherein the unlabeled data is only supervised loss training, and the labeled data is additionally class contrast learning loss training, and the most dissimilar instance feature in the labeled data is updated to the feature memory buffer; Finally, the student model is updated to the teacher model according to the exponential moving average method, wherein α=0.1; Step 7: Repeat steps 4 to 6 to reach the specified training steps, and apply the final teacher model to the test environment and make predictions.

2. The open-set semi-supervised image object detection method based on contrastive learning according to claim 1, characterized in that, In step 1, according to the number of instance level data in each category of labeled data, the instance level data of each category is mapped to the feature memory buffer by linear interpolation and scale factor mapping, and different size category feature memory buffers are adaptively allocated to each category instance level data. The category feature memory buffer mainly accesses the instance feature in the queue first-in-first-out manner in the data structure.

3. The open-set semi-supervised image object detection method based on contrastive learning according to claim 1, wherein, The labeled data is weakly enhanced in step 2, and the enhanced instance-level data is aligned with the label. The aligned data is sent to a Faster-RCNN model for supervised learning training. After training, the most similar instance-level data is updated to the feature memory buffer of the corresponding class according to the similarity distance The most dissimilar instance-level data is updated to the feature memory buffer of the corresponding class. The supervision loss is: (1) the RPN module classification loss, the RPN module regression loss, the ROI module classification loss, and the ROI module regression loss for target detection, which is used to improve the target detection accuracy; (2) the class contrast learning loss, which is used to reduce the intra-class distance of the same class label data and increase the inter-class distance of different class label data, so as to improve the feature extraction ability of the Faster-RCNN model and improve the classification accuracy of the Faster-RCNN model. The weights of the trained Faster-RCNN model are assigned to the teacher model and the student model. The teacher model and the student model have the same structure as the trained Faster-RCNN model, have the same weights, and have similar predictions. The class contrast learning loss is as follows: ； ； wherein, denotes an index subscript; denotes an index ; denotes the same class; denotes an indicator function, which is 0 when and 1 otherwise; denotes the number of instance features of class c; denotes the contrastive loss of instance features of class c and fixed anchor with index subscript i; denotes the contrastive loss of class c; denotes the number of other instance features with the same class as the fixed anchor with index subscript i; is the temperature coefficient.

4. The open-set semi-supervised image object detection method based on contrastive learning according to claim 1, characterized in that, The step 5 wherein , and are obtained from the following equations, respectively. ； ； ； wherein represents an instance feature, represents an instance feature corresponding to a pseudo label to be screened, represents an instance feature in a feature memory buffer of the same category as the pseudo label to be screened, represents an instance feature that obtains a minimum value in a plurality of dot multiplication operations as an anchor point , represents a different instance feature in the same category feature memory buffer, represents obtaining a maximum value in a plurality of dot multiplication operations as a threshold value , represents a dot multiplication operation of an instance feature corresponding to a pseudo label to be screened and an instance feature corresponding to an anchor point.

5. The open-set semi-supervised image object detection method based on contrastive learning according to claim 1, characterized in that, In step 5, the two filtering methods are dynamically controlled according to the voting method in ensemble learning, and any one of the two filtering methods determines that the pseudo label is true before N times the number of labeled data iterations, and is retained; If the iteration number exceeds, both methods need to determine that the pseudo label is true, and the pseudo label is retained.

Images