Open set semi-supervised target detection method based on class contrast learning

Abstract

The invention belongs to the field of computer vision, and discloses an open set semi-supervised target detection method based on class contrast learning. According to the method, a solution is provided for the instance-level approximate distribution external data problem, an approximate distribution external data filtering strategy is provided by combining a clustering thought and a class contrast learning thought, a dynamic boundary is abstractly divided for each prediction instance, and error semantic extension caused by approximate distribution external data instances is prevented. Through effective integration of a dynamic threshold strategy and class contrast learning, the situation that excessive normal instances are misjudged as out-of-distribution instances by the model due to excessive strict out-of-distribution exception filtering at the initial stage of training, and then the generalization performance of the model is affected can be prevented; and the generalization ability of the model can be improved by using normal instances in unmarked data and a small amount of approximate distribution external data instances, and label distribution can be strictly controlled in the later stage of training, so that the situation that the model performance is rapidly lowered due to overlarge data variance caused by the fact that normal pseudo labels are distributed to excessive approximate distribution external instances is prevented.

Description

Technical Field

[0001] The present invention belongs to target detection in the field of computer vision, and relates to an open-set semi-supervised target detection method with inconsistent data label space. Background Art

[0002] With the rapid development of deep learning technology, many researchers have adopted this advanced data-driven technology to achieve performance comparable to or even better than humans in many tasks in the field of computer vision. However, the outstanding results of deep learning rely heavily on high-quality data annotation. In order to reduce the time-consuming and labor-intensive annotation burden, deep semi-supervised learning has gradually become the research focus of many researchers because it only requires a small amount of labeled data. Among them, labeled data is also called in-domain data (ID) data. In recent years, deep semi-supervised learning has been widely studied in image classification and target detection tasks. In classification tasks, for example, Dash (Xu Y, Shang L, Ye J, et al. Dash: Semi-supervised learning with dynamic thresholding [C] / / International conference on machine learning. PMLR, 2021: 11525-11536.) first considered using dynamic thresholds to more reasonably filter pseudo-labels when screening pseudo-labels. In target detection tasks, for example, Unbiased teacher (Liu YC, Ma CY, He Z, et al. Unbiased teacher for semi-supervised object detection [J]. arXiv preprint arXiv: 2102.09480, 2021.) uses the teacher model and the student model to learn from each other, providing a basic framework for the subsequent development of the open-set semi-supervised target detection field.

[0003] Although the development of existing deep semi-supervised learning has significantly reduced the algorithm's dependence on ID data, most of these studies are based on a core assumption - ID samples and unlabeled samples follow the same data distribution, that is, the independent and identically distributed assumption. However, this assumption is not always true in the open environment of the real world. Specifically, the presence of a large amount of out-of-domain (OOD) data in unlabeled datasets often causes existing semi-supervised learning algorithms to produce incorrect semantic extensions, thereby seriously reducing model performance.

[0004] To this end, many researchers have considered solving the OOD problem in the deep semi-supervised learning framework to screen pseudo-labels, thereby reducing the expansion of erroneous semantics. Among them, in semi-supervised classification tasks, such as OpenMatch (Saito K, Kim D, Saenko K. Openmatch: Open-set semi-supervised learning with open-set consistency regularization [J]. Advances in Neural Information Processing Systems, 2021, 34: 25956-25967.) Two classifiers were trained using labeled data, one is a traditional classifier used to determine the category to which the data belongs, and the other is a One vs All classifier (Saito K, Saenko K. Ovanet: One-vs-all network for universal domain adaptation [C] / / Proceedings of the ieee / cvf international conference on computer vision. 2021: 9000-9009.), which aims to distinguish one category from other categories and assign pseudo-labels through the cooperation of the two classifiers. CCSSL (Yang F, Wu K, Zhang S, et al. Class-aware contrastive semi-supervised learning [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 14421-14430.) introduces a class-aware contrastive learning method to improve performance in open set settings. UnMixMatch (Roy S, Etemad A. Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38 (13): 14847-14856.) is different from the practice of filtering OOD data and then assigning pseudo labels. Instead, it highly integrates the idea of ​​self-supervised learning with semi-supervised learning, making full use of unlabeled data to enhance the model's ability to extract features. These studies have expanded the ideas for target detection tasks.

[0005] In the semi-supervised object detection task, there are also related studies to solve the OOD problem. For example, OSSOD (Liu YC, Ma CY, Dai X, et al. Open-set semi-supervised object detection [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 143-159.) adds an additional plug-and-play Dino pre-trained module to the basic framework of semi-supervised object detection, and considers using background category instances as OOD categories to fine-tune the Dino module to assist in filtering OOD instances. Subsequently, OOSSOD (Wang Z, Xiao L, Xiang L, et al. Online Open-set Semi-supervised Object Detection with Dual Competing Head [J]. arXiv preprint arXiv: 2305.13802, 2023.) proposed an end-to-end online framework, which borrowed the idea of ​​noisy labels and trained a pair of dual competitive classification heads to filter OOD instances.

[0006] In the present invention, we found that in the OOD data of the target detection task, in addition to the common OOD data with a large gap with the ID data, there is also a type of out-of-distribution data with a certain similarity to the labeled data, namely the near-domain out-of-distribution data (NearOOD). To distinguish the NearOOD data, we call the common OOD data FarOOD. Although the NearOOD data are not included in the ID data set, they have certain similarities with the ID data, so they are difficult to be recognized and detected by the visual processing model, which will cause the model to perform poorly due to excessive data variance. Yang et al. (Yang J, Zhou K, Liu Z. Full-spectrum out-of-distribution detection.arXiv 2022[J].arXiv preprintarXiv:2204.05306.) have specifically explored the Near OOD problem. However, the existing open set semi-supervised target detection method has not yet considered the problem of Near OOD, resulting in incorrect semantic generalization due to ignoring this part of the data, which in turn leads to a decrease in model detection performance. To this end, the present invention proposes to deeply explore the unique properties of Near OOD instances in the open set semi-supervised target detection task, and adaptively affect the iterative process of the target detection model. In the initial stage of the iteration of model training, the similarity between Near OOD instances and conventional data is used to enhance the model's ability to learn features. As the iteration deepens, since the model gradually builds up a strong learning ability, in the later stage of the iteration, it turns to a more clear OOD detection to assist the model in effectively identifying and eliminating Near OOD, so that clearer sample characteristics can be learned, thereby avoiding potential interference of NearOOD instances on the model prediction accuracy. Therefore, the present invention further improves the accuracy of open set semi-supervised target detection by solving the NearOOD problem. Summary of the invention

[0007] In order to solve the problem that the existence of Near OOD data interferes with the performance of open set semi-supervised object detection, this paper proposes an improved OSSOD framework based on class contrast learning. By more effectively using ID data to filter NearOOD data, the model can effectively learn accurate features, thereby improving the detection accuracy of the model. First, this paper adopts a class contrast learning strategy. By narrowing the similarity distance of instances of the same category and widening the similarity distance of instances of different categories, the inter-class boundary of ID data is clearer, thereby significantly improving the utilization efficiency of labeled data. In this way, the model can more accurately distinguish different categories of ID instances and enhance its adaptability to complex data distribution. In addition, in order to prevent the negative impact of erroneous semantic extension on model performance, this paper draws on the clustering idea and drives it using the similarity distance of class contrast learning. Specifically, this study stores a very small number of diverse instances for each ID category to simulate the cluster boundary of each category. In order to reduce the amount of calculation and consider the particularity of the open set, we do not use the traditional clustering method to find the center point. We use the ID instance closest to the predicted instance as the anchor, and take the similarity distance between the anchor and the most diverse ID instance in the same category as the threshold. In this way, we abstractly implement a dynamic threshold for each category or even each instance, thereby more effectively improving the out-of-distribution detection performance.

[0008] Finally, this paper proposes an integrated learning strategy for dynamically regulating the OOD filtering mechanism. This strategy can prevent the model from misjudging too many ID instances as OOD instances due to overly strict OOD filtering in the early stage of training, thereby affecting the generalization performance of the model. It can also strictly control label allocation in the later stage of training to prevent too many Near OOD instances from being assigned ID pseudo-labels, which causes the model performance to decline rapidly due to excessive data variance.

[0009] The technical solution of the present invention:

[0010] An open-set semi-supervised object detection method based on class contrastive learning, the steps are as follows:

[0011] Step 1: Adaptively allocate feature memory buffers of different sizes according to the instance-level data of each category of label data;

[0012] According to the number of instance-level data of each category in the labeled data, the instance-level data of each category is mapped to the feature memory buffer through linear interpolation and scale factor mapping, and a category feature memory buffer of different sizes is adaptively allocated to each category instance-level data. The category feature memory buffer mainly stores the labeled instance features in a first-in-first-out manner;

[0013] Step 2: Send the labeled data to the Faster-RCNN model for supervised learning training. At the same time, class contrast learning is used to shorten the intra-class distance and increase the inter-class distance. After iteration, the memory buffer of each category is updated.

[0014] The labeled data is weakly enhanced and the enhanced instance-level data is aligned with the label. The aligned data is sent to the Faster-RCNN model for supervised learning training and after training, the similarity distance is calculated. The least similar instance-level data of the same category is updated to the feature memory buffer of the corresponding category; the supervision loss is: (1) RPN module classification loss, RPN module regression loss, ROI module classification loss, and ROI module regression loss for target detection, which are used to improve the accuracy of target detection; (2) class contrast learning loss, which improves the ability of the Faster-RCNN model to extract features and improves the classification accuracy of the Faster-RCNN model by shortening the intra-class distance of the same category label data and increasing the inter-class distance of the different category label data; the weights of the trained Faster-RCNN model are assigned to the teacher model and the student model. The teacher model and the student model have the same structure as the trained Faster-RCNN model, so that they have the same weights and similar predictions. The class contrast learning loss is as follows:

[0015]

[0016] Among them, i, j, k represent index subscripts; z i Indicates that the index is i; Indicates that k and i belong to the same category; represents the indicator function, which is 0 when j=i and 1 otherwise; N represents the number of instance features of category c; L c,i represents the contrast loss of the instance feature with category c and fixed anchor index i; L c represents the contrast loss of category c; P(i) represents the number of other instance features of the same category as the fixed anchor with index subscript i; τ is the temperature coefficient;

[0017] Step 3: Repeat step 2 for 2K iterations, and after the iteration is completed, copy the weights to the teacher model and the student model, and both share the same weights;

[0018] Step 4: After weak data enhancement, the unlabeled data is sent to the teacher model for prediction to obtain the pseudo labels to be screened;

[0019] Step 5: Adaptively screen the generated pseudo-labels to be screened. Before a certain number of iterations, if any of the two screening mechanisms determines that the pseudo-label is true, it is true; after a certain number of iterations, both screening mechanisms need to determine that the pseudo-label is true at the same time to be true;

[0020] The pseudo labels to be screened are adaptively screened. The screening mechanism is two filtering methods that are dynamically regulated by the voting method and the number of iterations in the ensemble learning. The first filtering method is: compare the predicted value of the instance feature corresponding to the pseudo label to be screened with the dynamic threshold, and discard the instance features and pseudo labels below the dynamic threshold; the dynamic threshold is the initial value of 0.3, and increases to 0.8 within a certain round according to the linear function; the second filtering method is: find the instance feature that is most similar to the current predicted instance in the feature memory buffer corresponding to the pseudo label to be screened, and record it as the anchor point z anchor ; Then, the similarity distance between the anchor point and the least similar instance feature in the feature memory buffer of the corresponding category is used as the threshold SIM threshold ; If the similarity distance SIM between the instance feature corresponding to the current pseudo label to be filtered and the anchor point {pred,anchor} If z is greater than the threshold, the pseudo label of the current predicted instance is retained, otherwise it is discarded; anchor , SIM threshold and SIM {pred,anchor} They are obtained by the following formulas:

[0021]

[0022]

[0023] SIM {pred,anchor} =z pred ·z anchor

[0024] Where z represents the instance feature, z pred represents the instance features corresponding to the pseudo-label to be filtered, z l Represents the instance features in the feature memory buffer of the same category as the pseudo-label to be filtered. Minimize() represents the instance feature z that obtains the minimum value of multiple dot multiplication operations. l As anchor point z anchor , z l1 ,z l2 Represents different instance features in the same category feature memory buffer. Max() means obtaining the maximum value of multiple point multiplication operations as the threshold SIM threshold , SIM {pred,anchor} Represents the dot product operation of the instance feature corresponding to the pseudo label to be filtered and the instance feature corresponding to the anchor point;

[0025] The two filtering methods are dynamically regulated according to the voting method in ensemble learning. If any of the two filtering methods determines that the pseudo-label is true before the number of iterations N times the number of label data, both are retained; if the number of iterations exceeds this number, both methods need to determine that the pseudo-label is true at the same time to retain the pseudo-label;

[0026] Step 6: After assigning pseudo labels to the instance-level data corresponding to the unlabeled data, the pseudo labels are sent to the student model for training together with the labeled data. The unlabeled data is only trained with supervision loss, and the labeled data is additionally trained with class contrast learning loss. The most dissimilar instance features in the labeled data are updated to the feature memory buffer. Finally, the student model updates the teacher model according to the exponential moving average method, where α = 0.1.

[0027] Step 7: Repeat steps 4 to 6 to reach the specified number of training steps, apply the final teacher model to the test environment and make predictions.

[0028] Beneficial effects of the present invention: In the scenario of semi-supervised target detection, the present invention uses a basic dynamic threshold and a class contrast learning driven method to screen samples outside the approximate distribution, and realizes an end-to-end trainable open set semi-supervised target detection framework. For the first time, a solution is proposed for the problem of instance-level approximate out-of-distribution data. By combining the clustering idea and the class contrast learning idea, an innovative approximate out-of-distribution data filtering strategy is proposed, which abstractly divides the dynamic boundary for each prediction instance, thereby effectively preventing the erroneous semantic expansion caused by the approximate out-of-distribution data instance. And through the effective integration of the basic dynamic threshold strategy and class contrast learning, it can prevent excessively strict out-of-distribution anomaly filtering in the early stage of training, causing the model to misjudge too many normal instances as out-of-distribution instances, thereby affecting the generalization performance of the model, and can use normal instances in unlabeled data and a small number of approximate out-of-distribution data instances to improve the generalization ability of the model. It can also strictly control the label allocation in the later stage of training to prevent too many approximate out-of-distribution instances from being assigned normal pseudo-labels, causing the model performance to decline rapidly due to excessive data variance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A detailed training framework structure diagram of the open set semi-supervised target detection based on class contrastive learning provided by the present invention;

[0030] Figure 2 A schematic diagram of the division of the feature memory buffer provided by the present invention;

[0031] Figure 3 This is a schematic diagram of pseudo-label filtering based on similarity distance provided by the present invention. DETAILED DESCRIPTION

[0032] The following further describes the specific implementation of the present invention in combination with the accompanying drawings and technical solutions.

[0033] The present invention provides an open-set semi-supervised target detection method based on class contrast learning, and the specific implementation process is as follows: first, a small feature memory buffer is opened in the memory, and the total memory size is N times the number of labeled data categories, and N is generally 16; in order to allocate feature memory buffers of different sizes to each category, after counting the number of instance features of each category in the labeled data, linear interpolation and a proportional factor are used to calculate the size of the feature memory buffer of each category, wherein the size of the feature memory buffer is an even number, and the instance features are accessed in a queue first-in-first-out manner in a data structure; Figure 2 After allocating the feature memory buffers for each category, two-stage training begins. Figure 1 As shown in the figure, first, the labeled data is weakly enhanced with random data and then sent to the Faster-RCNN model for a certain round of supervised loss training, and when performing class contrast learning, the instance features with the same category but the least similar in a batch of labeled data are stored in the feature memory buffer of the corresponding category. After the first stage of training is completed, the weights of the trained Faster-RCNN model are assigned to the teacher model and the student model. The teacher model and the student model have the same structure as the trained Faster-RCNN model, so that they have the same weights and similar predictions, and the second stage of training begins; in the second stage of training, the unlabeled data is firstly weakly enhanced with random data and then sent to the teacher model for prediction, and the instance features in the unlabeled data are assigned their prediction labels, also called pseudo labels. After obtaining the pseudo-label, it is necessary to filter it. The filtering mechanism is that before a certain number of iterations, two filtering methods are required to jointly determine that the pseudo-label predicted by the instance feature is wrong, and the pseudo-label will be discarded. After a certain number of iterations, if any filtering method determines that the pseudo-label predicted by the instance feature is wrong, the pseudo-label will be discarded. The two filtering methods are as follows: (1) Dynamic threshold filtering method, set the initial threshold to 0.3, set the final threshold to 0.8, and use the linear function Where n is the total number of iterations. When the maximum predicted value of the instance feature is less than the dynamic threshold, the pseudo-label corresponding to the instance feature is judged to be wrong. (2) A pseudo-label filtering method based on similarity distance is used to find the labeled instance feature that is most similar to the current instance feature in the feature memory buffer corresponding to the category of the instance feature pseudo-label, and record it as the anchor point z. anchor Then, we will find the instance feature that is most dissimilar to the anchor point in the feature memory buffer of the same category, and calculate the similarity distance between the anchor point and it as the threshold SIM threshold , if the similarity distance between the instance feature currently to be determined and the anchor point is SIM {pred,anchor} If the value is less than the threshold, the pseudo-label corresponding to the instance feature to be screened is judged as wrong. Figure 3 shown.

[0034] After filtering the pseudo-labels, the pseudo-labels are assigned to the unlabeled data, and the pseudo-labeled data and the labeled data are strongly enhanced by random data and sent to the student model for supervised loss learning, where the labeled data is subjected to class comparison learning, and the instance features of the same category and the least similar are also stored in the corresponding feature memory buffer. After a training session, the weights of the student model are updated to the teacher model through exponential moving average. After a certain number of rounds of repeated training, the weights of the teacher model are fixed and applied to the test environment.

[0035] In general, the present invention solves the problem of Near OOD instance recognition through an open set semi-supervised target detection model with dynamic regulation and class contrast learning, mainly through class contrast learning and voting mechanism in ensemble learning. Specifically, the combination of dynamic threshold and voting mechanism not only reduces the complexity of setting hyperparameters, but also effectively improves the robustness of the model through a staged regulation mechanism, which not only avoids the performance degradation of the model due to error accumulation, but also can use Near OOD for learning to a certain extent.

Claims

1. An open-set semi-supervised object detection method based on class contrastive learning, characterized in that: Here are the steps: Step 1: Adaptively allocate feature memory buffers of different sizes according to the instance-level data of each category of label data; According to the number of instance-level data of each category in the labeled data, the instance-level data of each category is mapped to the feature memory buffer through linear interpolation and scale factor mapping, and a category feature memory buffer of different sizes is adaptively allocated for each category instance-level data. The category feature memory buffer mainly accesses instance features in a first-in-first-out manner in the queue in the data structure; Step 2: Send the labeled data to the Faster-RCNN model for supervised learning training. At the same time, class contrast learning is used to shorten the intra-class distance and increase the inter-class distance. After iteration, the memory buffer of each category is updated. The labeled data is weakly enhanced and the enhanced instance-level data is aligned with the label. The aligned data is sent to the Faster-RCNN model for supervised learning training and after training, the similarity distance is calculated. The least similar instance-level data of the same category is updated to the feature memory buffer of the corresponding category; the supervision loss is: (1) RPN module classification loss, RPN module regression loss, ROI module classification loss, and ROI module regression loss for target detection, which are used to improve target detection accuracy; (2) class contrast learning loss, which improves the ability of the Faster-RCNN model to extract features and improves the classification accuracy of the Faster-RCNN model by shortening the intra-class distance of the same category label data and increasing the inter-class distance of the different category label data; the weights of the trained Faster-RCNN model are assigned to the teacher model and the student model. The teacher model and the student model have the same structure as the trained Faster-RCNN model, so that they have the same weights and similar predictions. The class contrast learning loss is as follows: Among them, i, j, k represent index subscripts; z i Indicates that the index is i; Indicates that k and i belong to the same category; represents the indicator function, which is 0 when j=i and 1 otherwise; N represents the number of instance features of category c; L c,i represents the contrast loss of the instance feature with category c and fixed anchor index i; L c represents the contrast loss of category c; P(i) represents the number of other instance features of the same category as the fixed anchor with index subscript i; τ is the temperature coefficient; Step 3: Repeat step 2 for 2K iterations, and after the iteration is completed, copy the weights to the teacher model and the student model, and both share the same weights; Step 4: After weak data enhancement, the unlabeled data is sent to the teacher model for prediction to obtain the pseudo labels to be screened; Step 5: Adaptively screen the generated pseudo-labels to be screened. Before a certain number of iterations, if any of the two screening mechanisms determines that the pseudo-label is true, it is true; after a certain number of iterations, both screening mechanisms need to determine that the pseudo-label is true at the same time to be true; The pseudo labels to be screened are adaptively screened. The screening mechanism is two filtering methods that are dynamically regulated by the voting method and the number of iterations in the ensemble learning. The first filtering method is: compare the predicted value of the instance feature corresponding to the pseudo label to be screened with the dynamic threshold, and discard the instance features and pseudo labels below the dynamic threshold; the dynamic threshold is the initial value of 0.3, and increases to 0.8 within a certain round according to the linear function; the second filtering method is: find the instance feature that is most similar to the current predicted instance in the feature memory buffer corresponding to the pseudo label to be screened, and record it as the anchor point z anchor ; Then, the similarity distance between the anchor point and the least similar instance feature in the feature memory buffer of the corresponding category is used as the threshold SIM threshold ; If the similarity distance SIM between the instance feature corresponding to the current pseudo label to be filtered and the anchor point {pred,anchor} If z is greater than the threshold, the pseudo label of the current predicted instance is retained, otherwise it is discarded; anchor , SIM threshold and SIM {pred,anchor} They are obtained by the following formulas: SIM {pred,anchor} =from pred ·With anchor Where z represents the instance feature, z pred represents the instance features corresponding to the pseudo-label to be filtered, z l Represents the instance features in the feature memory buffer of the same category as the pseudo-label to be filtered. Minimize() represents the instance feature z that obtains the minimum value of multiple dot multiplication operations. l As anchor point z anchor , z l1 ,z l2 Represents different instance features in the same category feature memory buffer. Max() means obtaining the maximum value of multiple point multiplication operations as the threshold SIM threshold , SIM {pred,anchor} Represents the dot product operation of the instance feature corresponding to the pseudo label to be filtered and the instance feature corresponding to the anchor point; The two filtering methods are dynamically regulated according to the voting method in ensemble learning. If any of the two filtering methods determines that the pseudo-label is true before the number of iterations N times the number of label data, both are retained; if the number of iterations exceeds this number, both methods need to determine that the pseudo-label is true at the same time to retain the pseudo-label; Step 6: After assigning pseudo labels to the instance-level data corresponding to the unlabeled data, the pseudo labels are sent to the student model for training together with the labeled data. The unlabeled data is only trained with supervision loss, and the labeled data is additionally trained with class contrast learning loss. The most dissimilar instance features in the labeled data are updated to the feature memory buffer. Finally, the student model updates the teacher model according to the exponential moving average method, where α = 0.

1. Step 7: Repeat steps 4 to 6 to reach the specified number of training steps, apply the final teacher model to the test environment and make predictions.

Images