A semi-supervised object detection system based on distinguishable features and its training method
Through a semi-supervised object detection system based on distinguishable features, we use distinguishable feature heat maps and contrastive clustering loss functions to generate target box proposals and perform data enhancement, which solves the problem of insufficient utilization of unlabeled data in semi-supervised target detection and improves detection accuracy and efficiency.
Patent Information
- Application Number
- CN202211422460.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-14
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2042-11-14
AI Technical Summary
In existing semi-supervised object detection methods, the model over-relies on labeled data and under-utilizes unlabeled data, resulting in low detection efficiency.
A semi-supervised object detection system based on distinguishable features is adopted. Through the input image feature extraction module, the distinguishable feature sensitive region proposal module, the distinguishable feature extraction and storage module and the distinguishable feature data enhancement module, the distinguishable feature heat map and the contrast clustering loss function are used to generate target box proposals and perform data enhancement to improve the utilization rate of unlabeled data.
It improves the model's utilization of unlabeled data, enhances the accuracy and efficiency of target detection, and solves the problem of insufficient utilization of unlabeled data in semi-supervised learning.
Smart Images
Figure CN115797621B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target detection in machine vision, and in particular to a semi-supervised object detection system based on distinguishable features and a training method thereof. Background Art
[0002] Deep learning has achieved tremendous success in recent years. However, deep learning algorithms have a huge demand for data. Although a large number of open-source, implementable deep learning algorithms are available for practical applications, achieving high accuracy, this still relies on users building their own datasets based on specific scenarios, which incurs significant costs in both money and manpower. In the field of object detection, compared to simpler visual tasks such as image classification, which only require labeling the object category in an image, labeling object detection datasets requires not only classifying a single object instance but also labeling the locations of four boundary points. The labeling time for a single object is approximately 10 seconds. This is especially true in complex scenarios such as autonomous driving, where the number of objects is greater and the labeling difficulty is enormous. Therefore, considering that obtaining unlabeled data is much easier than obtaining labeled data, exploring semi-supervised object detection methods that use a small amount of labeled data and a large amount of unlabeled data to train object detectors has become an important research issue.
[0003] While semi-supervised learning methods have been widely used in image classification tasks, semi-supervised object detection is still in its early stages of research. A major obstacle limiting its development is that, compared to image classification, the input data for object detection may contain multiple object instances. This requires the model to not only classify each instance but also output the specific location of each instance. Existing mainstream semi-supervised methods fall into two categories: pseudo-label-based methods and consistency regularization-based methods. Pseudo-label-based methods primarily use labeled data to first train a classifier to output pseudo-labels for unlabeled data. Finally, the model is trained on unlabeled data with the pseudo-labels. Consistency regularization-based methods rely on the principle that model predictions should remain unchanged even after adding noise to unlabeled images. Therefore, these methods encourage the addition of perturbations or other data augmentation techniques to the unlabeled samples, ensuring that the model predictions remain unchanged before and after the perturbations. However, while both of these methods have proven effective, they still have limitations: pseudo-labeling methods cannot address the problem of error accumulation caused by pseudo-label prediction errors, while consistency regularization methods rely too heavily on labeled data, yielding little useful information from the unlabeled data. How to maximize the use of labeled data and make as much use of unlabeled data as possible becomes the main challenge of semi-supervised learning.
[0004] Humans are able to distinguish objects with minimal data. This is primarily due to their ability to discover and summarize distinguishable features in images. After learning from a small number of examples, they can easily identify the location and category of an instance in an image based on key features. For example, humans can confidently infer the category of an instance based on a small area of leopard print. However, neural networks excel at generalizing biases across large inputs and lack the ability to summarize distinguishable features. This may be the main reason for their low learning efficiency. Therefore, enabling models to summarize distinguishable features can better summarize useful information from small amounts of labeled data. It can also enable models to capture effective information related to distinguishable features from unlabeled data, thereby improving detector accuracy. Summary of the Invention
[0005] In view of this, the purpose of the present invention is to propose a semi-supervised object detection system based on distinguishable features and a training method thereof, so as to solve the problem in the prior art of over-reliance on labeled data and insufficient utilization of unlabeled data in the semi-supervised learning process.
[0006] Based on the above objectives, the present invention provides a semi-supervised object detection system based on distinguishable features, the system comprising:
[0007] Input image feature extraction module: used to extract features of input image data and encode the features into high-dimensional features;
[0008] A distinguishable feature-sensitive region proposal module is configured to generate a distinguishable feature heat map of the input image data based on the high-dimensional features, and to assist the region proposal network in positioning through the distinguishable feature heat map to generate a target box proposal;
[0009] Distinguishing feature extraction and storage module: used to extract distinguishable features from the target frame proposal, compare and cluster the extracted distinguishable features to form various distinguishable feature sets, and then store the prototype vectors of various distinguishable feature sets;
[0010] The distinguishable feature data enhancement module is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data, forcing the system to continuously explore new distinguishable features in the target proposal;
[0011] An object recognition and localization module is configured to predict the correct object category and location using the distinguishable features and features in the target proposal.
[0012] Furthermore, in the distinguishable feature-sensitive region proposal module, a class activation map is selected as a discrimination criterion for distinguishable features, and a threshold of the class activation map is set to filter high-dimensional features to generate a distinguishable feature heat map.
[0013] Furthermore, in the distinguishable feature-sensitive region proposal module, during the process of the region proposal network generating the target box proposal, the mean of the category activation map within the anchor is calculated as a weighted loss function term, so that the target box proposal generated by the region proposal network contains as many distinguishable features as possible.
[0014] Furthermore, in the distinguishable feature extraction and storage module, a contrast clustering loss is introduced to bring the distances between features of the same class closer and to keep the distances between features of different classes farther apart, thereby forming distinguishable feature sets of each class.
[0015] Furthermore, the distinguishable feature set of the class is defined as the prototype vector of the distinguishable features of each class extracted: P = {p0, p1...p c};
[0016] Among them, p is the prototype vector; c is the number of categories;
[0017] Use contrast loss to cluster distinguishable feature vectors to increase the difference between classes. The formula of the loss function used is:
[0018]
[0019]
[0020] Among them, p i is the prototype vector of category i, and the prototype vector of the category is the average value of the feature vectors of each extracted distinguishable feature;
[0021] D is the Euclidean distance function;
[0022] Δ is the distance parameter between similar and dissimilar classes;
[0023] f c is the cth distinguishable feature of the input;
[0024] l u (f c ) is the total loss function for all categories;
[0025] l(f c ,p i ) is the loss function corresponding to the i-th target;
[0026] D(f c ,p i ) is f c and p i distance.
[0027] Furthermore, in the distinguishable feature data enhancement module, the mean value of the high-dimensional feature is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data.
[0028] Here, we also introduce a training method for a semi-supervised object detection system based on distinguishable features. The training method is as follows:
[0029] Step 1: Determine the target detection network and build the system as described above;
[0030] Step 2: Collect application scenario datasets, which include labeled data and unlabeled data;
[0031] Step 3: Use the application scenario dataset to train the system;
[0032] Step 4: Iterate the previous step until the trained system model achieves the ideal object detection effect.
[0033] Furthermore, the target detection network adopts the network structure of Faster-RCNN.
[0034] Furthermore, according to the system model constructed in step 1, the order in which the input image data is propagated in the system model is the order in which the system model is constructed, and the construction order is as follows:
[0035] Step 101: Output high-dimensional features of input image data through an input image feature extraction module;
[0036] Step 102: Input the high-dimensional features into the distinguishable feature-sensitive region proposal module to obtain the distinguishable feature heat map and target box proposal.
[0037] Step 103: Input the target box proposal into the distinguishable feature extraction and storage module to obtain the distinguishable features in the target box proposal and store the prototype vectors of various distinguishable feature sets;
[0038] Step 104: For the unlabeled data in the input image, input it into the distinguishable feature data enhancement module to continuously obtain new distinguishable features;
[0039] Step 105: Input all distinguishable features and features in the target proposal into the object recognition and positioning module to predict the correct object category and location.
[0040] Furthermore, the training method of step 3 is as follows:
[0041] Stage S1: Use labeled data to train the system, calculate and store the prototype vectors of distinguishable features in each category of labeled data; to distinguish the systems in each stage, this stage system is represented as System 1.
[0042] Stage S2: Input unlabeled data into system 1 to generate corresponding target box proposals, category pseudo labels and distinguishable feature heat maps for each input image data.
[0043] Stage S3: Copy System 1 to System 2, use the distinguishable feature data enhancement module to perform data enhancement on the unlabeled data with random masking according to the distinguishable feature heat map output in the previous stage, and then use the pseudo-label and unlabeled data to train System 2; after the training iteration, use System 2 to update System 1.
[0044] Beneficial effects of the present invention: The semi-supervised object detection system based on distinguishable features and the training method thereof of the present invention have the following advantages:
[0045] 1. This paper proposes a method for forcing models to learn more efficiently by identifying distinguishable features. This method uses distinguishable feature detection in semi-supervised object detection to help the model summarize key knowledge points from labeled data and discover richer and more effective information from unlabeled data.
[0046] 2. This paper proposes a feature-sensitive region proposal network to address the inaccurate positioning of target bounding boxes in semi-supervised object detection. It uses the detected distinguishable feature locations to assist the region proposal network in generating target bounding box proposals. Based on this, a distinguishable feature detector is constructed to generate and store distinguishable feature prototype vectors for each category, thereby outputting more accurate target bounding box classification results.
[0047] 3. This paper proposes a data augmentation method that randomly blocks distinguishable features. This method addresses the problem of related methods over-relying on the quality of labeled data detectors and under-utilizing unlabeled data. By randomly blocking some distinguishable feature regions in the input unlabeled data, the model is encouraged to learn more effective feature information from the unlabeled data. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0049] Figure 1 Schematic diagram of the process of the semi-supervised object detection system based on distinguishable features of the present invention;
[0050] Figure 2 Flowchart of a training method for a semi-supervised object detection system based on distinguishable features of the present invention;
[0051] Figure 3 A flowchart for training the system using application scenario datasets. DETAILED DESCRIPTION
[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.
[0053] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.
[0054] In the first aspect of the present invention, a semi-supervised object detection system based on distinguishable features is proposed, such as Figure 1 As shown, the system includes:
[0055] Input image feature extraction module: used to extract features of input image data and encode the features into high-dimensional features;
[0056] A distinguishable feature-sensitive region proposal module is configured to generate a distinguishable feature heat map of the input image data based on the high-dimensional features, and to assist the region proposal network in positioning through the distinguishable feature heat map to generate a target box proposal;
[0057] Distinguishing feature extraction and storage module: used to extract distinguishable features from the target frame proposal, compare and cluster the extracted distinguishable features to form various distinguishable feature sets, and then store the prototype vectors of various distinguishable feature sets;
[0058] The distinguishable feature data enhancement module is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data, forcing the system to continuously explore new distinguishable features in the target proposal;
[0059] An object recognition and localization module is configured to predict the correct object category and location using the distinguishable features and features in the target proposal.
[0060] In this embodiment, the distinguishable features are the main features existing in the image that affect the system's judgment of the object, such as special texture details, special contours, etc.
[0061] As an implementation method, in the distinguishable feature-sensitive region proposal module, a class activation map is selected as a criterion for distinguishable features, a threshold of the class activation map is set to filter high-dimensional features, and a distinguishable feature heat map is generated.
[0062] In this example, the Class Activation Map (CAM) is used as the criterion for distinguishable features. The output value of the CAM is larger for regions with important distinguishable features, while the output value is smaller for regions with no obvious background features. By setting a threshold on the CAM, regions with distinguishable features can be filtered out, thereby helping the region proposal network generate more accurate object box proposals.
[0063] As an embodiment, in the distinguishable feature-sensitive region proposal module, during the process of the region proposal network generating a target box proposal, the mean of the category activation map within the anchor is calculated as a weighted loss function term, so that the target box proposal generated by the region proposal network contains as many distinguishable features as possible.
[0064] As an implementation method, in the distinguishable feature extraction and storage module, a contrast clustering loss is introduced to make the distance between features of the same class closer and the distance between features of different classes farther apart, thereby forming a distinguishable feature set of each class for subsequent data enhancement and classification. The specific details are as follows: The distinguishable feature set of a class is defined as the prototype vector of the distinguishable features of each class extracted: P = {p0, p1...p c};
[0065] Among them, p is the prototype vector; c is the number of categories;
[0066] Use contrast loss to cluster distinguishable feature vectors to increase the difference between classes. The formula of the loss function used is:
[0067]
[0068]
[0069] Among them, p i is the prototype vector of category i, and the prototype vector of the category is the average value of the feature vectors of each extracted distinguishable feature;
[0070] D is the Euclidean distance function;
[0071] Δ is the distance parameter between similar and dissimilar classes;
[0072] f c is the cth distinguishable feature of the input;
[0073] l u (f c ) is the total loss function for all categories;
[0074] l(f c ,p i ) is the loss function corresponding to the i-th target;
[0075] D(f c ,p i ) is f c and p i distance.
[0076] Here, by extracting and storing the prototype vectors of distinguishable features, the network can make better use of labeled information and use the labeled information for subsequent classification of unlabeled data, making the classification results more accurate.
[0077] As an implementation manner, in the distinguishable feature data enhancement module, the mean value of the high-dimensional feature is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data.
[0078] Specifically, while using the trained model to generate pseudo-labels for unlabeled data, the system also generates a discriminative feature heatmap for each unlabeled image to visualize the areas of high density of discriminative features that the model focuses on. During the training phase using unlabeled data, the discriminative feature data enhancement module randomly masks discriminative features using the mean of the entire dataset, forcing the model to focus on features in other discriminative areas.
[0079] The overall training framework of this paper follows the teacher-student mechanism widely used in semi-supervised learning. First, a teacher model is trained using labeled data. Then, a student model is trained using unlabeled data and pseudo-labels generated by the teacher model. After a period of time, the student model is used to update the teacher model parameters. Furthermore, distinguishable features are incorporated as auxiliary training to fully utilize the data features.
[0080] Here, we also introduce a training method for a semi-supervised object detection system based on distinguishable features, such as Figure 2 As shown, the training method is as follows:
[0081] Step 1: Determine the target detection network and build the system as described above;
[0082] Step 2: Collect application scenario datasets, which include labeled data and unlabeled data;
[0083] Step 3: Use the application scenario dataset to train the system;
[0084] Step 4: Iterate the previous step until the trained system model achieves the ideal object detection effect.
[0085] As an implementation method, the target detection network adopts the network structure of Faster-RCNN.
[0086] As an implementation, according to the system model constructed in step 1, the order in which the input image data is propagated in the system model is the order in which the system model is constructed, and the construction order is as follows:
[0087] Step 101: Output high-dimensional features of input image data through an input image feature extraction module;
[0088] Step 102: Input the high-dimensional features into the distinguishable feature-sensitive region proposal module to obtain the distinguishable feature heat map and target box proposal.
[0089] Step 103: Input the target box proposal into the distinguishable feature extraction and storage module to obtain the distinguishable features in the target box proposal and store the prototype vectors of various distinguishable feature sets;
[0090] Step 104: For the unlabeled data in the input image, input it into the distinguishable feature data enhancement module to continuously obtain new distinguishable features;
[0091] Step 105: Input all distinguishable features and features in the target proposal into the object recognition and positioning module to predict the correct object category and location.
[0092] As an implementation method, Figure 1 、 Figure 3 As shown, the training method of step 3 is as follows:
[0093] Stage S1: Use labeled data to train the system, calculate and store the prototype vectors of distinguishable features in each category of labeled data; to distinguish the systems in each stage, this stage system is represented as System 1.
[0094] Stage S2: Input unlabeled data into system 1 to generate corresponding target box proposals, category pseudo labels and distinguishable feature heat maps for each input image data.
[0095] Stage S3: Copy System 1 to System 2, use the distinguishable feature data enhancement module to perform data enhancement on the unlabeled data with random masking according to the distinguishable feature heat map output in the previous stage, and then use the pseudo-label and unlabeled data to train System 2; after the training iteration, use System 2 to update System 1.
[0096] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Within the scope of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.
[0097] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A semi-supervised object detection system based on distinguishable features, characterized in that: The system comprises: Input image feature extraction module: used to extract features of input image data and encode the features into high-dimensional features; A distinguishable feature-sensitive region proposal module is configured to generate a distinguishable feature heat map of the input image data based on the high-dimensional features, and to assist the region proposal network in positioning through the distinguishable feature heat map to generate a target box proposal; Distinguishing feature extraction and storage module: used to extract distinguishable features from the target frame proposal, compare and cluster the extracted distinguishable features to form various distinguishable feature sets, and then store the prototype vectors of various distinguishable feature sets; The distinguishable feature data enhancement module is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data, forcing the system to continuously explore new distinguishable features in the target proposal; An object recognition and localization module is configured to predict the correct object category and location using the distinguishable features and features in the target proposal.
2. A semi-supervised object detection system based on distinguishable features according to claim 1, characterized in that: In the distinguishable feature-sensitive region proposal module, a class activation map is selected as a criterion for distinguishable features, and a threshold of the class activation map is set to filter high-dimensional features to generate a distinguishable feature heat map.
3. A semi-supervised object detection system based on distinguishable features according to claim 2, characterized in that: In the distinguishable feature-sensitive region proposal module, during the process of the region proposal network generating the target box proposal, the mean of the category activation map within the anchor is calculated as a weighted loss function term, so that the target box proposal generated by the region proposal network contains as many distinguishable features as possible.
4. A semi-supervised object detection system based on distinguishable features according to claim 3, characterized in that: In the distinguishable feature extraction and storage module, a contrast clustering loss is introduced to make the distances between features of the same class closer and the distances between features of different classes farther apart, thereby forming distinguishable feature sets of each class.
5. A semi-supervised object detection system based on distinguishable features according to claim 4, characterized in that: The distinguishable feature set of the definition class is the prototype vector of the distinguishable features of each category extracted: ; in, p is the prototype vector; c is the number of categories; Use contrast loss to cluster distinguishable feature vectors to increase the difference between classes. The formula of the loss function used is: in, p i For category i The prototype vector of the category is the average value of the feature vectors of each extracted distinguishable feature; D is the Euclidean distance function; Δ is the distance parameter between similar and dissimilar classes; f c For the input c distinguishable features; l u ( f c ) is the total loss function for all categories; l ( f c , p i ) is the i The loss function corresponding to the class target; D( f c , p i )for f c and p i distance.
6. A semi-supervised object detection system based on distinguishable features according to claim 5, characterized in that: In the distinguishable feature data enhancement module, the mean value of the high-dimensional feature is used to mask some distinguishable features in the target box proposal generated for the unlabeled input image data.
7. A training method for a semi-supervised object detection system based on distinguishable features, characterized in that: The training method is as follows: Step 1: Determine a target detection network and build a system as described in any one of claims 1 to 6; Step 2: Collect application scenario datasets, which include labeled data and unlabeled data; Step 3: Use the application scenario dataset to train the system; Step 4: Iterate the previous step until the trained system model achieves the ideal object detection effect.
8. The method for training a semi-supervised object detection system based on distinguishable features according to claim 7, wherein: The target detection network adopts the network structure of Faster-RCNN.
9. The method for training a semi-supervised object detection system based on distinguishable features according to claim 8, characterized in that: According to the system model constructed in step 1, the order in which the input image data is propagated in the system model is the order in which the system model is constructed, and the construction order is as follows: Step 101: Output high-dimensional features of input image data through an input image feature extraction module; Step 102: Input the high-dimensional features into the distinguishable feature-sensitive region proposal module to obtain the distinguishable feature heat map and target box proposal; Step 103: Input the target box proposal into the distinguishable feature extraction and storage module to obtain the distinguishable features in the target box proposal and store the prototype vectors of various distinguishable feature sets; Step 104: For the unlabeled data in the input image, input it into the distinguishable feature data enhancement module to continuously obtain new distinguishable features; Step 105: Input all distinguishable features and features in the target proposal into the object recognition and positioning module to predict the correct object category and location.
10. The method for training a semi-supervised object detection system based on distinguishable features according to claim 9, wherein: The training method of step 3 is as follows: Stage S1: Use labeled data to train the system, calculate and store the prototype vectors of distinguishable features in each category of labeled data; to distinguish the systems in each stage, this stage is denoted as System 1; Stage S2: Input unlabeled data into system 1 and generate corresponding target box proposals, category pseudo labels and distinguishable feature heat maps for each input image data; Stage S3: Copy System 1 to System 2, use the distinguishable feature data augmentation module to perform data augmentation on the unlabeled data with random masking based on the distinguishable feature heat map output in the previous stage, and then train System 2 with pseudo-labeled and unlabeled data; after the training iteration, use System 2 to update System 1.
Citation Information
Patent Citations
Spatial resolution enhancement method based on weak supervision deep learning, terminal equipment and computer readable storage medium
CN112862809A
Semi-supervised image analysis method for power transmission scene
CN115272777A