Semi-supervised target detection method based on open vocabulary target detection technology
By introducing open vocabulary object detection technology and large-scale unsupervised self-coding model into the object detection method, efficient object detection in large-scale data is achieved, solving the problem of poor detection performance of existing methods under large data distribution changes, and improving the generalization ability and robustness of the model.
Patent Information
- Application Number
- CN202411749331.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-25
AI Technical Summary
The existing target detection methods have poor detection performance in large-scale data, especially when the data distribution changes greatly, and the equipment demand is high, making it difficult to widely use.
The semi-supervised object detection method based on open vocabulary object detection technology is adopted to pre-label the image data through preset prompt words, and a feature library is built using a large unsupervised self-coding model, and compared it with the pre-label detection box, filtering pseudo-labels, and combining manual labeling data for mixed training of proprietary models.
It improves data utilization, reduces labeling resource loss, improves the generalization ability and robustness of proprietary models, and ensures the accuracy and real-time detection.
Smart Images
Figure QLYQS_1 
Figure QLYQS_2 
Figure QLYQS_3
Abstract
Description
Technical Field
[0001] The present invention is applied to the field of computer vision technology, and specifically is a semi-supervised target detection method based on open vocabulary target detection technology. Background Art
[0002] As a traditional task in the field of computer vision, object detection has always occupied a very important position in real-world applications such as safe production, autonomous driving, and face recognition. However, with the increasing number of real-world application scenarios and the explosive growth of image data, the location search and annotation of effective data under huge data volumes have become a bottleneck that limits the growth of real-world application scenarios of object detection. Therefore, how to effectively utilize huge data and reduce the human cost is an urgent problem to be solved. In recent years, with the rapid development of deep learning, data as the original driving force is an unavoidable threshold, while data annotation and cleaning are very time-consuming and labor-intensive. In order to solve the problem that existing object detection methods can only detect in limited categories, open-vocabulary object detection (OVD) effectively expands the categories that existing object detection methods can detect by fusing information from two modalities, text and image. However, due to the influence of text feature alignment, when the existing data distribution is significantly different from the original training data distribution, the detection performance in specific tasks is difficult to reach the existing proprietary model capabilities, and especially its real-time performance is poor and occupies a large number of resources, so it has high equipment requirements and is difficult to be widely used in real-world applications. Considering its inherent openness, applying it to the search and pre-labeling of large-scale data and generating corresponding pseudo-labels to assist in the training of proprietary models can greatly improve the utilization of data and reduce the resource loss related to labeling in data engineering. By making full use of data, the generalization ability and robustness of proprietary models can be improved.
[0003] At present, the actual application workflow of target detection tasks can be roughly divided into three stages: data labeling, model training, and fine-tuning and optimization. Data labeling accounts for about 25% or more of the overall development process, and the larger the amount of data, the greater the positive effect of labeling on improving the generalization ability and robustness of the model. However, a large amount of data will increase the labeling time, making it impossible to advance the development task at normal time. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a semi-supervised target detection method based on open vocabulary target detection technology in view of the deficiencies in the prior art.
[0005] In order to solve the above technical problems, a semi-supervised target detection method based on open vocabulary target detection technology of the present invention specifically comprises the following steps:
[0006] Preset relevant prompt words, input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain pre-labeled detection boxes;
[0007] Build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library;
[0008] The pre-labeled detection boxes are encoded using a large unsupervised autoencoding model and compared with the corresponding category features in the feature library, and the results with comparison results below the preset threshold are filtered out;
[0009] Select a proprietary model network for training and use pre-labeled data and manually labeled data for mixed training;
[0010] The accuracy and recall rates are calculated in the test set according to different detection thresholds, and threshold screening is performed.
[0011] As a possible implementation, further, the steps of presetting relevant prompt words, inputting relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain pre-labeled detection boxes are specifically as follows:
[0012] For video data, use a preset frequency to extract images and unify the extracted images to the same resolution;
[0013] Image data, uniform resolution, set prompt words according to the category to be detected, the model obtains the area with high similarity based on the input image and text prompt, and returns it as the detection box. The similarity calculation formula is as follows:
[0014]
[0015] Where Q and C represent the total number of interest regions and the number of input prompts obtained each time, and R and T represent the high-dimensional feature representation of the interest region and the text high-dimensional feature representation of the prompt, respectively.
[0016] As a possible implementation, further, the typical sample library is constructed according to the category to be detected, and a large unsupervised autoencoder model is used to extract features of the images in the sample library. The steps of constructing a corresponding feature library according to the sample library specifically include:
[0017] According to the categories that need to be detected, a preset number of related images are manually annotated for each category, and the annotated box part is cropped.
[0018] As a possible implementation, further, the typical sample library is constructed according to the category to be detected, and features of the images in the sample library are extracted using a large unsupervised autoencoder model. The step of constructing a corresponding feature library according to the sample library also includes:
[0019] A large unsupervised autoencoder model is used to encode the cropped images, store the obtained relevant features, and build a feature library. The unsupervised autoencoder model is mainly trained using the contrastive learning method, and the loss function is as follows:
[0020]
[0021] Where q and k represent the image features used as queries and key values in the contrastive learning training process, respectively. T represents the temperature coefficient, which is used to control the smoothness of the distribution between the positive and negative sample sets. + represents the positive sample after data enhancement.
[0022] As a possible implementation, further, the steps of encoding the pre-labeled detection frame using a large unsupervised autoencoding model, comparing it with the corresponding category features in the feature library, and filtering the results whose comparison results are lower than a preset threshold are specifically as follows:
[0023] The same large-scale unsupervised autoencoder model as that used to construct the feature library is used to extract features from the pre-labeled detection frames. At the same time, the pre-labeled detection frames are compared with the samples in the category feature library of the corresponding category, and the cosine similarity between the pre-labeled frames and the samples in the feature library is calculated. The pre-labeled frames with a cosine similarity threshold below the threshold are filtered out according to the pre-set cosine similarity threshold.
[0024] As a possible implementation, further, the selecting of a proprietary model network for training and performing mixed training using pre-labeled data and manually labeled data specifically includes:
[0025] Select a proprietary model network for training and perform data preprocessing on the pre-labeled data and manually labeled data according to the network input.
[0026] As a possible implementation, further, the selecting of a proprietary model network for training and performing mixed training using pre-labeled data and manually labeled data also includes:
[0027] Design a data loading module, perform average sampling on pre-labeled data and manually labeled data, and reconstruct the loss function. Different weights are assigned according to the input data type, mainly including classification loss and regression loss, and the output detection box and category are optimized respectively. The loss function is as follows:
[0028] L reg = -log(IOU(Bgt ,B dt )) α -log(IOU(B gt ,B dt )) β
[0029]
[0030] Where IOU is expressed as the overlapping area between the calculated true box and the predicted detection box, α and β represent the weights for the pre-labeled box and the manually labeled box respectively, the overall loss is composed of regression and classification losses, B represents the coordinates of the detection box, and p represents the predicted probability of the final output of the model.
[0031] A semi-supervised object detection system based on open vocabulary object detection technology, specifically comprising:
[0032] The pre-labeled detection frame generation module is used to preset relevant prompt words, input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain the pre-labeled detection frame;
[0033] The feature library construction module is used to build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library;
[0034] The pre-labeled detection frame filtering module is used to encode the pre-labeled detection frame using a large unsupervised autoencoding model, and compare it with the corresponding category features in the feature library, and filter the results whose comparison results are lower than the preset threshold;
[0035] A proprietary model training module is used to select a proprietary model network for training and perform mixed training using pre-labeled data and manually labeled data;
[0036] The threshold screening and performance evaluation module is used to perform threshold screening based on the calculation results of accuracy and recall rate in the test set according to different detection thresholds.
[0037] The present invention adopts the above technical solution and has the following beneficial effects:
[0038] 1. By adopting the open vocabulary object detection method to pre-label large quantities of unlabeled data, the data utilization rate is improved, and the error rate of pseudo-labeling is reduced by feature matching.
[0039] 2. Fuse manually annotated images and pseudo-labeled images to train the proprietary model, and improve the robustness of the proprietary model by increasing the number of training sets.
[0040] 3. By evaluating the model performance and determining the final threshold on an independent test set, the stability and accuracy of the alarm can be guaranteed in different situations. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.
[0042] Example 1
[0043] The present invention provides a semi-supervised target detection method based on open vocabulary target detection technology, which specifically comprises the following steps:
[0044] Preset relevant prompt words, input relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain pre-labeled detection frames; for video data, use the preset frequency to extract images, and unify the extracted images to the same resolution;
[0045] Image data, uniform resolution, set prompt words according to the category to be detected, the model obtains the area with high similarity based on the input image and text prompt, and returns it as the detection box. The similarity calculation formula is as follows:
[0046]
[0047] Where Q and C represent the total number of interest regions and the number of input prompts obtained each time, and R and T represent the high-dimensional feature representation of the interest region and the text high-dimensional feature representation of the prompt, respectively.
[0048] A typical sample library is built according to the categories that need to be detected, and a large unsupervised autoencoding model is used to extract features of the images in the sample library. A corresponding feature library is built based on the sample library. According to the categories that need to be detected, a preset number of related images are manually labeled for each category, and the labeled box part is cropped.
[0049] A large unsupervised autoencoder model is used to encode the cropped images, store the obtained relevant features, and build a feature library. The unsupervised autoencoder model is mainly trained using the contrastive learning method, and the loss function is as follows:
[0050]
[0051] Where q and k represent the image features used as queries and key values in the contrastive learning training process, respectively. T represents the temperature coefficient, which is used to control the smoothness of the distribution between the positive and negative sample sets. + represents the positive sample after data enhancement.
[0052] The pre-labeled detection frame is encoded using a large unsupervised autoencoder model and compared with the corresponding category features in the feature library, and the results with comparison results below a preset threshold are filtered; the pre-labeled detection frame is extracted using the same large unsupervised autoencoder model as used to construct the feature library, and it is compared with samples in the category feature library of the corresponding category, and the cosine similarity between the pre-labeled frame and the samples in the feature library is calculated, and the pre-labeled frames below the threshold are filtered according to a pre-set cosine similarity threshold;
[0053] Select a proprietary model network for training, and use pre-labeled data and manually labeled data for mixed training; that is, select a proprietary model network for training, and pre-process the pre-labeled data and manually labeled data according to the network input. Design a data loading module, sample the pre-labeled data and manually labeled data on average, and reconstruct the loss function at the same time, assign different weights according to the input data type, mainly including classification loss and regression loss, and optimize the output detection box and category respectively. The loss function is as follows:
[0054] L reg = -log(IOU(B gt ,B dt )) α -log(IOU(B gt ,B dt )) β
[0055]
[0056] Where IOU is expressed as the overlapping area between the calculated true box and the predicted detection box, α and β represent the weights for the pre-labeled box and the manually labeled box respectively, the overall loss is composed of regression and classification losses, B represents the coordinates of the detection box, and p represents the predicted probability of the final output of the model.
[0057] The accuracy and recall rates are calculated in the test set according to different detection thresholds, and threshold screening is performed.
[0058] A semi-supervised object detection system based on open vocabulary object detection technology, specifically comprising:
[0059] The pre-labeled detection frame generation module is used to preset relevant prompt words, input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain the pre-labeled detection frame;
[0060] The feature library construction module is used to build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library;
[0061] The pre-labeled detection frame filtering module is used to encode the pre-labeled detection frame using a large unsupervised autoencoding model, and compare it with the corresponding category features in the feature library, and filter the results whose comparison results are lower than the preset threshold;
[0062] A proprietary model training module is used to select a proprietary model network for training and perform mixed training using pre-labeled data and manually labeled data;
[0063] The threshold screening and performance evaluation module is used to perform threshold screening based on the calculation results of accuracy and recall rate in the test set according to different detection thresholds.
[0064] Example 2
[0065] The OVD method is used to pre-label a large number of unlabeled data through text prompts, and the relevant images of the required categories are sorted to construct a feature library. The pre-trained image feature encoder is used to compare the features of the large number of pseudo labels generated with the images in the feature library, and the pseudo labels below the set threshold are filtered out. The remaining pseudo labels and the labeled labels are used for data fusion training.
[0066] The specific steps are as follows:
[0067] Step 1: Set a set of relevant prompt words according to the detection task, and input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection.
[0068] Step 1.1: For video data, use a frequency of 1 frame per second to extract images, and unify the extracted images to the same resolution. The image data is directly unified in resolution. Set the prompt word according to the category to be detected, for example: a car driving at high speed, a bicycle, etc. The model can obtain the area with high similarity based on the input image and text prompt, and return it as the detection box. The similarity calculation formula is as follows:
[0069]
[0070] Where Q and C represent the total number of regions of interest (ROI) and the number of input prompts obtained each time. R and T represent the high-dimensional feature representation of the region of interest and the high-dimensional feature representation of the prompt text, respectively.
[0071] Step 2: Build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library.
[0072] Step 2.1: Manually label 20 to 30 relevant images for each category according to the categories to be detected, and crop the labeled box part
[0073] Step 2.2: Use a large unsupervised autoencoder model to encode the cropped part of the image, store the obtained relevant features, and build a feature library. The unsupervised autoencoder model is mainly trained using the contrastive learning method, and the loss function is as follows:
[0074]
[0075] Where q and k represent the image features used as queries and key values in the contrastive learning training process, respectively. The purpose is usually to ensure that the dot product on the diagonal of the image feature dot product matrix is maximized. T represents the temperature coefficient, which is used to control the smoothness of the distribution between the positive and negative sample sets, and + represents the positive sample after data enhancement.
[0076] Step 3: Encode the pre-labeled detection box obtained in step 1 using a large unsupervised autoencoder model and compare it with the corresponding category features in the feature library. Filter the results below the pre-set filter threshold.
[0077] Step 3.1: Use the same large-scale unsupervised autoencoder model as that used to construct the feature library to extract features from the pre-labeled detection box, and compare it with the samples in the category feature library of the corresponding category, calculate the cosine similarity between the pre-labeled box and the samples in the feature library, and filter the pre-labeled boxes below the cosine similarity threshold according to the pre-set threshold;
[0078] Step 4: Select a proprietary model network for training and mix the pre-labeled data and manually labeled data for training.
[0079] Step 4.1: Select a proprietary model network for training and perform data preprocessing on the pre-labeled data and manually labeled data according to the network input.
[0080] Step 4.2: Design a data loading module, perform average sampling on pre-labeled data and manually labeled data, and reconstruct the loss function. Different weights are assigned according to the input data type, mainly including classification loss and regression loss, and the output detection box and category are optimized respectively. The loss function is as follows:
[0081] L reg = -log(IOU(B gt ,B dt )) α -log(IOU(B gt ,B dt )) β(3)
[0082]
[0083] Where IOU is expressed as the overlapping area between the calculated true box and the predicted detection box, α and β represent the weights for the pre-labeled box and the manually labeled box respectively, the overall loss is composed of regression and classification losses, B represents the coordinates of the detection box, and p represents the predicted probability of the final output of the model.
[0084] Step 5: Evaluate the performance of the model on a test set with a different domain from the training set and determine the final model threshold to ensure performance while maintaining a high user experience.
[0085] Step 5.1 uses different detection thresholds to calculate the accuracy and recall rate in the test set, and selects the threshold with relatively balanced accuracy and recall rate and good performance as the threshold for the final online application.
[0086] The above are embodiments of the present invention. For ordinary technicians in the field, according to the teachings of the present invention, all equivalent changes, modifications, substitutions and variations made within the scope of the patent application of the present invention without departing from the principles and spirit of the present invention should fall within the scope of the present invention.
Claims
1. A semi-supervised object detection method based on open vocabulary object detection technology, characterized in that: The specific steps include: Preset relevant prompt words, input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain pre-labeled detection boxes; Build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library; The pre-labeled detection boxes are encoded using a large unsupervised autoencoding model and compared with the corresponding category features in the feature library, and the results with comparison results below the preset threshold are filtered out; Select a proprietary model network for training and use pre-labeled data and manually labeled data for mixed training; The accuracy and recall rates are calculated in the test set according to different detection thresholds, and threshold screening is performed.
2. The semi-supervised target detection method based on open vocabulary target detection technology according to claim 1, characterized in that: The steps of presetting relevant prompt words and inputting relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain pre-labeled detection frames are specifically as follows: For video data, use a preset frequency to extract images and unify the extracted images to the same resolution; Image data, uniform resolution, set prompt words according to the category to be detected, the model obtains the area with high similarity based on the input image and text prompt, and returns it as the detection box. The similarity calculation formula is as follows: Where Q and C represent the total number of interest regions and the number of input prompts obtained each time, and R and T represent the high-dimensional feature representation of the interest region and the text high-dimensional feature representation of the prompt, respectively.
3. The semi-supervised target detection method based on open vocabulary target detection technology according to claim 1, characterized in that: The typical sample library is constructed according to the categories to be detected, and features of the images in the sample library are extracted using a large unsupervised autoencoder model. The steps of constructing a corresponding feature library according to the sample library specifically include: According to the categories that need to be detected, a preset number of related images are manually annotated for each category, and the annotated box part is cropped.
4. The semi-supervised target detection method based on open vocabulary target detection technology according to claim 1, characterized in that: The typical sample library is constructed according to the categories to be detected, and features of the images in the sample library are extracted using a large unsupervised autoencoder model. The steps of constructing a corresponding feature library according to the sample library also include: A large unsupervised autoencoder model is used to encode the cropped images, store the obtained relevant features, and build a feature library. The unsupervised autoencoder model is mainly trained using the contrastive learning method, and the loss function is as follows: Where q and k represent the image features used as queries and key values in the contrastive learning training process, respectively. T represents the temperature coefficient, which is used to control the smoothness of the distribution between the positive and negative sample sets. + represents the positive sample after data enhancement.
5. The semi-supervised target detection method based on open vocabulary target detection technology according to claim 1, characterized in that: The steps of encoding the pre-labeled detection frame using a large unsupervised autoencoding model, comparing it with the corresponding category features in the feature library, and filtering the results whose comparison results are lower than a preset threshold are specifically as follows: The same large-scale unsupervised autoencoder model as that used to construct the feature library is used to extract features from the pre-labeled detection boxes. At the same time, they are compared with the samples in the category feature library of the corresponding category, and the cosine similarity between the pre-labeled boxes and the samples in the feature library is calculated. The pre-labeled boxes that are below the cosine similarity threshold are filtered out according to the pre-set cosine similarity threshold.
6. The semi-supervised target detection method based on open vocabulary target detection technology according to claim 1, characterized in that: The method of selecting a proprietary model network for training and performing mixed training using pre-labeled data and manually labeled data specifically includes: Select a proprietary model network for training and perform data preprocessing on the pre-labeled data and manually labeled data according to the network input.
7. The semi-supervised object detection method based on open vocabulary object detection technology according to claim 1, characterized in that: The selecting of a proprietary model network for training and performing mixed training using pre-labeled data and manually labeled data also includes: Design a data loading module, perform average sampling on pre-labeled data and manually labeled data, and reconstruct the loss function. Different weights are assigned according to the input data type, mainly including classification loss and regression loss, and the output detection box and category are optimized respectively. The loss function is as follows: L reg =-log(IOU(B gt ,B dt )) α -log(IOU(B gt ,B dt )) β Where IOU is expressed as the overlapping area between the calculated true box and the predicted detection box, α and β represent the weights for the pre-labeled box and the manually labeled box respectively, the overall loss is composed of regression and classification losses, B represents the coordinates of the detection box, and p represents the predicted probability of the final output of the model.
8. A semi-supervised object detection system based on open vocabulary object detection technology, characterized in that: Specifically include: The pre-labeled detection frame generation module is used to preset relevant prompt words, input the relevant prompt words and pre-processed image data into the pre-trained OVD network for detection to obtain the pre-labeled detection frame; The feature library construction module is used to build a typical sample library according to the categories that need to be detected, use a large unsupervised autoencoding model to extract features from the images in the sample library, and build a corresponding feature library based on the sample library; The pre-labeled detection frame filtering module is used to encode the pre-labeled detection frame using a large unsupervised autoencoding model, and compare it with the corresponding category features in the feature library, and filter the results whose comparison results are lower than the preset threshold; A proprietary model training module is used to select a proprietary model network for training and perform mixed training using pre-labeled data and manually labeled data; The threshold screening and performance evaluation module is used to perform threshold screening based on the calculation results of accuracy and recall rate in the test set according to different detection thresholds.