Semi-supervised object detection method and platform based on pseudo-gain category rebalancing

Abstract

The present invention discloses a semi-supervised target detection method and platform based on pseudo-gain category rebalancing, the method comprising: establishing a training data set S0, the training data set S0 comprising: labeled samples and a plurality of unlabeled samples; performing supervised learning based on the labeled samples to obtain a target detector M0; and i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i The pseudo gain score of each unlabeled sample in the training dataset S i Select K unlabeled samples for manual labeling to obtain the training data set S i+1 ; After all unlabeled samples are labeled, obtain the target detector M; perform target detection on the image to be detected based on the target detector M. The present invention obtains a data set with balanced distribution and rich information content based on the sample mining strategy of pseudo gain, so as to maximize the utilization of information in the unlabeled data.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and relates to a semi-supervised target detection method and platform based on pseudo-gain category rebalancing. Background Art

[0002] It is becoming a general trend to apply the research results of computer vision based on deep learning to production data processing. The steady growth of the number of netizens and the continuous innovation of Internet business models have led to an explosive growth in the size of pictures on social platforms. These data sets with rich information can help the network learn distinguishable features. Among them, target detection is used to solve the problem of positioning and classification of targets in images. It can not only directly obtain the semantic information of the surface of image information, but also assign it to the corresponding downstream tasks for further analysis, such as face clustering, scene recognition, etc. Target detection algorithms are usually divided into single-stage or double-stage. The difference lies in whether the generation of candidate areas is used as an independent link. However, most mainstream detection algorithms are oriented to supervised learning and require a large amount of human-labeled data to improve the generalization of the model.

[0003] Active learning and semi-supervised learning provide a good solution to the labor cost and performance constraints caused by data labeling. Active learning prioritizes the original samples with the highest information value for labeling, and achieves the expected model performance with the least amount of labeled samples. Semi-supervised learning uses a small amount of labeled data and a large amount of unlabeled data, continues to apply the supervised learning paradigm to the labeled data, and gradually passes the information of the unlabeled data to the model.

[0004] Although active learning and semi-supervised learning are important pillars of deep learning, the development of related tools has received far less attention than related algorithm research. Existing annotation training tools or systems do not fully combine the advantages of active learning and semi-supervised learning.

[0005] In addition, faced with changes in target detection task requirements and the expansion of the range of targets to be detected, the existing active learning schemes have not fully considered the collection of new class instances. When data collection for new categories is insufficient and there is a huge disparity between the amount of data between the new and old categories, the model will tend to favor categories with a larger number of samples. Summary of the invention

[0006] In response to the above problems, the present invention discloses a semi-supervised target detection method and platform based on pseudo-gain category rebalancing, which aims to integrate the main ideas of active learning and semi-supervised learning and optimize the two strategies in a targeted manner, and finally realize step-by-step training by building an interactive system integrating data screening, labeling and training, thereby reducing technical and labor costs.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] A semi-supervised target detection method based on pseudo-gain category rebalancing, the method comprising:

[0009] Create a training dataset S 0 , the training data set S 0 Includes: labeled samples and several unlabeled samples;

[0010] Based on the labeled samples, supervised learning is performed to obtain the target detector M 0 ;

[0011] According to the target detector M i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i The pseudo gain score of each unlabeled sample in is obtained by manually labeling K unlabeled samples to obtain the training data set S i+1 , where the target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed, where i is the number of training rounds;

[0012] After all unlabeled samples are labeled, obtain the target detector M;

[0013] Object detection is performed on the image to be detected based on the object detector M.

[0014] Furthermore, the target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed, including:

[0015] Based on the training data set S i The labeled data in the target detector M i-1 Perform supervised training to obtain the target detector M′ i-1 ;

[0016] Copy the target detector M′ i-1 , get the target detector With object detector

[0017] Object Detector With object detector For the training data set S i The weak data enhancement version and the strong data enhancement version of each unlabeled sample in are forward calculated separately;

[0018] Object Detector The trained parameters are passed to the trained object detector via the exponential moving average method. To obtain the target detector M i .

[0019] Further, the target detector M i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i The pseudo gain scores of each unlabeled sample in include:

[0020] Based on the inference results, calculate the training data set S i The unlabeled samples in are selected and labeled as the training data set S i Resulting in pseudo-gains;

[0021] For the training data set S i The inference result of any unlabeled sample in is calculated based on the confidence level of each category, and the entropy of each instance in the inference result is calculated;

[0022] Based on the entropy of each instance, the entropy uncertainty score of the unlabeled sample is calculated;

[0023] Based on the pseudo gain and the entropy uncertainty score, a pseudo gain score of the unlabeled sample is calculated.

[0024] Furthermore, the method for calculating the entropy uncertainty score of the unlabeled sample includes: using a Max. aggregation function, a Min. aggregation function, an Avg. aggregation function or a Sum. aggregation function.

[0025] Furthermore, the method further comprises: obtaining a training data set S i+1 Afterwards, other unlabeled samples are used to train the training dataset S i+1 Amplify.

[0026] A semi-supervised target detection platform for implementing the above method comprises:

[0027] Front-end display;

[0028] Backend logic, including:

[0029] Training process management module, used to establish training data set S 0 , the training data set S 0 It includes: labeled samples and several unlabeled samples; supervised learning is performed based on the labeled samples to obtain a target detector M 0 ; According to the target detector M i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. iEach unlabeled sample in the training data set S i The pseudo gain of i Select K unlabeled samples for manual labeling to obtain the training data set S i+1 , where the target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed, where i is the number of training rounds; after all unlabeled samples are labeled, the target detector M is obtained;

[0030] The target detection module is used to perform target detection on the image to be detected based on the target detector M.

[0031] Furthermore, the background logic also includes a rights management module, which is used for background user and user rights management logic.

[0032] Furthermore, the interface of the training process management module includes: an instance creation class interface and a module control class interface;

[0033] The instance creation class interface is used to create a semi-supervised training control module instance and an active learning sample screening module instance;

[0034] The module control class interface is used for direct control of the active learning module and the semi-supervised learning module, and the communication control interface between the active learning module and the semi-supervised learning module.

[0035] Furthermore, the module control class interface includes: a policy modification class interface, a marking statistics and new addition class interface and a switching control class interface;

[0036] The strategy modification interface is used to modify the active learning strategy and the semi-supervised training strategy;

[0037] The annotation statistics and new class interface are used to analyze the training data set S i Statistics and annotation control interface of internal data;

[0038] The switching control class interface is used to switch whether the current process is in active learning sample screening or in a semi-supervised training process.

[0039] A storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.

[0040] Compared with the prior art, the present invention has at least the following advantages:

[0041] 1. This paper proposes a sample mining strategy based on pseudo-gain, which jointly determines the sample value by entropy uncertainty and pseudo-gain measurement. In each round of active learning experiments, this strategy can give high scores to images containing more insufficient category instances, and obtain a balanced and information-rich data set at a lower cost of data collection, screening, and annotation.

[0042] 2. The present invention proposes a learning ecosystem for massive unlabeled data - "training circle", which uses active learning and semi-supervised learning at the data level and model level respectively and integrates the two in series to form a cyclic training strategy, hoping to control the labor cost in the learning process while maximizing the utilization of information in the unlabeled data.

[0043] 3. The present invention designs and implements a deep learning platform for massive unlabeled images based on the training circle mechanism, embeds a variety of active learning strategies including pseudo-gain and the Unbiased Teacher semi-supervised learning framework. Compared with single functional encapsulation, the system controls the data flow cycle and can significantly reduce the time and energy cost of dual-line operations and switching between data set update and model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Insufficient class sample mining process based on pseudo gain.

[0045] Figure 2 The transformation process of unlabeled data in the training cycle.

[0046] Figure 3 The interface and calling structure of the semi-supervised object detection platform. DETAILED DESCRIPTION

[0047] In order to make the above features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention is further described below through specific embodiments.

[0048] 1. Sample mining strategy based on pseudo-gain

[0049] The semi-supervised target detection method of the present invention proposes a comprehensive measure "pseudo gain", which improves the mining of insufficient sample categories by introducing the concept of information gain and combining the measure of entropy uncertainty. The overall process is as follows: Figure 1 .

[0050] The current labeled dataset s will be used 0 The feature vector of the probability distribution of the i-th picture obtained by training the model is represented as Pr i (Y|θ;s 0 ), θ is the current model parameter, n is the number of sample categories, D represents the sample distribution of all currently labeled data, and D newIt represents the category distribution after adding the sample. Assuming that an unlabeled sample is selected and labeled, the pseudo gain it brings to the original data set is calculated as follows:

[0051]

[0052] The sample entropy uncertainty is calculated as follows:

[0053]

[0054] Combining the two, the comprehensive score of the sample is:

[0055]

[0056] w 0 is the reconciliation factor of the information gain part, which is a non-negative number and has two main purposes. One is to reconcile the decrease in the magnitude of information gain caused by the increase in the amount of labeled data, that is, for information gain:

[0057]

[0058] Since the probability is always less than 1, we have:

[0059]

[0060]

[0061] Among them, G is the information gain, G′ k is the derivative of the information gain G with respect to the current number of labeled samples k. From equations (5) and (6), we can see that the upper bound of the information gain depends on a function related to the current number of labeled samples k, and this function is -1 -1,+∞) increases monotonically, so a weight correction with gradually decreasing information gain is needed to maintain the discovery of insufficient samples.

[0062] Secondly, the category distribution of unlabeled images can only be calculated through the inference results of the detection model, which is not the true distribution of the image. The information gain calculated from this distribution is defined as "pseudo gain" in this paper, and its credibility is adjusted with the model accuracy. Linear growth is used as the reconciliation factor in this method. A smaller weight should be used in the early stage of training and then gradually increased.

[0063] The basic unit for evaluating and sorting information in the screening stage is the image. iIt is the score given to image sample i. However, it should be noted that the calculation method of pseudo gain naturally determines that the smallest unit considered is the image, but the smallest unit for entropy uncertainty calculation is the instance. When calculating the image entropy, the entropy of the many instances it contains must be aggregated. For image sample i, assuming that the model has a total of t instances in its reasoning results, for instance j, the entropy H can be calculated based on its confidence in each category. j If the Max. or Min. aggregation function is used, the instance with the largest or smallest entropy is selected from all instances, and the instance is used as the representative of the image. Its entropy score is used as the entropy uncertainty score. Finally, the pseudo gain obtained by formula (1) and the entropy uncertainty score are obtained by formula (2), and the pseudo gain score of the image is obtained by formula (3).

[0064] When calculating the entropy uncertainty score, if the Avg. or Sum. aggregation function is used, the average entropy or the sum of the entropies of all t instances is taken as the final score of the image. The Sum. function usually favors images with a large number of instances, because an image with a single high entropy uncertainty instance may score lower than an image with many low uncertainty objects.

[0065] 2. Learning ecosystem for massive unlabeled data

[0066] While conducting in-depth research on the screening and training of unlabeled image samples, the present invention aims to build a cyclic training strategy that integrates active learning and semi-supervised learning. The availability of unlabeled data is reflected in two stages. Stage one is used for high-quality sample mining driven by pseudo-gain, and the expansion of supervised data can be achieved after semi-automatic manual annotation; stage two is to participate in the training of semi-supervised target detection models without additional processing. After obtaining task-related image data, the working mode of the cyclic training strategy can be compared to the biosphere, which uses the nutrients of unlabeled images in segments and continuously transforms and promotes the flow of information.

[0067] Active learning that converts some images in unlabeled data into labeled data can be regarded as producers, and semi-supervised learning that uses newly generated labeled data together with the remaining unlabeled data for model learning can be regarded as consumers. The main role of consumers is to convert the "matter" of data into the "energy" of the model. Producers can provide consumers with high-information labeled samples to improve the contribution rate of supervisory information with limited manpower and time costs. Consumers will feed back the high-precision models and difficult samples produced by consumers to producers to assist producers in the second round of sample screening operations. In this way, they will jointly build an ecological cycle for unlabeled data learning and utilization. This process will be referred to as the "training circle" in the following text. Figure 2 Specifically, the following steps are included:

[0068] (1) If no labeled data is available, K images are randomly selected from the unlabeled data for manual labeling. If labeled data is available, this step is skipped.

[0069] (2) Use labeled data for supervised learning. First, extract the global features of the image through the backbone network, and then send them to the target detection model for category and positioning learning to obtain the initial detector Model. S ;

[0070] (3) Update the labeled dataset under active learning, which includes two steps: using the latest detector to infer the remaining unlabeled samples and manually labeling the samples of the top-K pseudo-gains;

[0071] (4) If there is still unlabeled data remaining, the semi-supervised training is performed using the labeled data set updated in step (3) and all unlabeled data sets. The open source framework Unbiased Teacher can be used for this purpose. In the warm-up phase, the labeled data is used for supervised training. The resulting model is copied to obtain two identical models, which are used as the Teacher and Student models respectively. In the teacher-student mutual learning phase, the Teacher model and the Student model perform forward calculations on the weak data enhancement and strong data enhancement versions of the same input respectively. The former generates a pseudo label after filtering or temperature scaling, and the latter is used as the output after perturbation to calculate the consistency loss with the pseudo label. After absorbing the knowledge of the unlabeled data, the Student model is passed to the Teacher model through the exponential moving average (EMA) to update the Teacher model and prevent the pseudo labels that are biased towards multiple sample categories from further exacerbating the impact of the imbalance of training data on model training. The semi-supervised mode makes the learning scope of the model no longer limited to the labeled images, and the learning objectives are no longer limited to the mapping relationship between images and labels. After the training is completed, jump to step (3);

[0072] (5) If there is no unlabeled data available, supervised learning is performed using the labeled data set updated in step (3) to obtain the final detection model and end the process.

[0073] 3. Deep learning platform for massive unlabeled images

[0074] The system realizes the mobilization and operation of the learning ecological cycle in the form of multi-user collaboration, and serves the relevant personnel of a certain target detection deep learning task, including project managers, deep learning engineers and annotation engineers. The main functions are as follows:

[0075] (1) For all users: system login and project opening, data set information overview;

[0076] (2) For deep learning engineers: Active learning strategy setting in high-information sample screening, parameter setting and monitoring of semi-supervised model training, training cycle startup, loop, save settings and process monitoring;

[0077] (3) For annotation engineers: semi-automatic interactive annotation;

[0078] (4) For project managers: project management, project group creation, and user management.

[0079] The implementation of the system is divided into two parts: front-end display and background logic. The front-end display part will be mainly written in the PyQt framework. The background logic has two main points from the perspective of the implemented functions. The first is the authority management module, which is used for the background user and user authority management logic; the second is the training process management module, which is based on the training control logic of active learning and semi-supervised training cycles. When the specific user roles and projects are determined, the overall training strategy will be fixed accordingly, and the project as a whole naturally complies with the object-oriented programming method. Therefore, this system encapsulates the project in an abstract class and instantiates the abstract class for different projects. In addition, there should be two important components in the logical structure of the abstract class of the project, namely, an instance for screening labeled samples for the active learning strategy and an instance for controlling the actual process under the semi-supervised training strategy. In addition, the background logic also includes a target detection module, which is used to detect targets based on the target detector M for the image to be detected.

[0080] The training process management module should include the following two types of interfaces:

[0081] (1) Instance creation class interface, including creating semi-supervised training control module instances and active learning sample screening module instances.

[0082] (2) Module control interface, including direct control of the active learning module and the semi-supervised learning module and the control interface for communication between the two modules.

[0083] like Figure 3 As shown, the present invention uses the response of the control signal of the front-end interface as the organizational basis, the arrow represents the call relationship, the dotted arrow represents the possible call relationship, and the overall interface is introduced along the different signal response interfaces. For the instance creation class interface of the project, in addition to creating it in the default way and directly calling the class constructor __init__ interface, more often use the load_from_json interface to load the existing configuration file, where the configuration file should include active learning (selecting samples with large information content from massive unlabeled samples for manual labeling) and semi-supervised learning configuration file paths and specific data paths and other basic information.

[0084] The module control interface includes three branches, namely, the strategy modification interface, the annotation statistics and new addition interface, and the switching control interface.

[0085] The policy modification interface includes two interfaces, change_AL and change_semi, which are used to modify the active learning policy and the semi-supervised training policy respectively. The two policy-level modification interfaces are logically similar, so they are introduced together. Taking the active learning part as an example, for the dynamically modifiable part, the attributes of the created instance are usually modified, such as the scoring standard, and then the modified content is written into the configuration file. For information that cannot be dynamically modified, such as the setting of the active learning backbone network structure, the corresponding configuration information of the active learning should be modified first, and then the active learning sample screening module should be created by the re-modified configuration file.

[0086] The labeling statistics and new class interfaces include get_data_info, add_label, and load_label interfaces, which are statistical and labeling control interfaces for the data in the current dataset. The main operation objects of this part of the interface are the labeling record files in the directory where the data is located.

[0087] The switching control interface includes the switch interface, which is mainly used to switch whether the current process is in active learning sample screening or in a semi-supervised training process. The network model is loaded synchronously during the switch. Since the two processes are logically mutually exclusive, the switching mode can save video memory usage without affecting functionality and fluency.

[0088] Experimental data

[0089] This experiment will use the detection data set collected from the Weibo platform, including 8 types of targets in two scenes, indoor and outdoor, to expand the detection scene to simulate the expansion of detection needs. Here, the targets in indoor scenes are taken as the original detection tasks, and the targets in outdoor scenes are taken as new categories. According to the active learning experiment process, a small number of images are first extracted for annotation to train the initial model, and then 5 rounds of iterations are performed, each round including the screening of data to be annotated, annotation, data set update, and model training. The comparison results are shown in Table 1.

[0090] The pseudo gain weight w in the first iteration 1=0, so the data collection and model training effects are similar to the results under the entropy measurement. Later, as the weight increases, it gradually shows a tendency towards the newly added categories. Observing the entropy measurement, it also showed a tendency towards the newly added class instances in the early iterations, but as the new and old sample sizes gradually tended to balance, there was no obvious tendency. The possible reason is that the detection difficulty of the newly added class instances in the early iteration experiments was higher, mainly due to the insufficient learning of the model for the newly added class instances. In the active learning algorithm based on entropy uncertainty, the higher the detection difficulty, the higher the entropy of the instance. Entering the later stage of the iteration, the model's detection and classification capabilities for the newly added categories are improved, and the focus is re-placed on difficult samples whose features are very likely to confuse the model, and no longer pays attention to the sample category. At the same time, with the improvement of model accuracy, the credibility of the pseudo-gain of paying attention to the category distribution is improved, and the expected utility is played. On the premise of ensuring the quality of the samples, more attention is paid to the categories with missing samples.

[0091] Table 1 Comparison of model training effects after enabling pseudo gain

[0092]

[0093] In summary, the semi-supervised target detection method of the present invention proposes an active learning method based on pseudo-gain measurement, which can achieve rebalancing of category distribution during the expansion of the data set, and starts from the two levels of data and model. A learning ecological "training circle" that maximizes the utilization of unlabeled data information is proposed, with training circle mobilization and multi-user collaboration as the main functional requirements of the system, realizing a deep learning platform for massive unlabeled images.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or replace the technical solutions of the present invention with equivalents. The protection scope of the present invention shall be subject to the claims.

Claims

1. A semi-supervised target detection method based on pseudo-gain category rebalancing, the method comprising: Establishing a training data set S0, wherein the training data set S0 includes: a labeled sample and a plurality of unlabeled samples; Perform supervised learning based on the labeled samples to obtain a target detector M0; According to the target detector M i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i The pseudo gain score of each unlabeled sample in is obtained by manually labeling the K unlabeled samples and obtaining the training data set S i+1 , wherein the target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed to obtain, i is the number of training rounds, and the target detector M i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i The pseudo gain scores of each unlabeled sample in include: Based on the inference results, calculate the training data set S i The unlabeled samples in are selected and labeled as the training data set S i Resulting in pseudo-gains; For the training data set S i The inference result of any unlabeled sample in is calculated based on the confidence level of each category, and the entropy of each instance in the inference result is calculated; Based on the entropy of each instance, the entropy uncertainty score of the unlabeled sample is calculated; Calculating a pseudo gain score of the unlabeled sample based on the pseudo gain and the entropy uncertainty score; After all unlabeled samples are labeled, obtain the target detector M; Object detection is performed on the image to be detected based on the object detector M.

2. The method according to claim 1, characterized in that The target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed, including: Based on the training data set S i The labeled data in the target detector M i-1 Perform supervised training to obtain the target detector M′ i-1 ; Copy the target detector M′ i-1 , get the target detector With object detector Object Detector With object detector For the training data set S i The weak data enhancement version and the strong data enhancement version of each unlabeled sample in are forward calculated separately; Object Detector The trained parameters are passed to the trained object detector via the exponential moving average method. To obtain the target detector M i .

3. The method according to claim 1, characterized in that The method for calculating the entropy uncertainty score of the unlabeled sample includes: using a Max. aggregation function, a Min. aggregation function, an Avg. aggregation function or a Sum. aggregation function.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: obtaining a training data set S i+1 Afterwards, other unlabeled samples are used to train the training dataset S i+1 Amplify.

5. A target detection platform for implementing the method according to any one of claims 1 to 4, comprising: Front-end display; Backend logic, including: The training process management module is used to establish a training data set S0, wherein the training data set S0 includes: a labeled sample and a plurality of unlabeled samples; supervised learning is performed based on the labeled samples to obtain a target detector M0; and i For the training data set S i The inference results of the unlabeled samples in the training dataset S are calculated. i Each unlabeled sample in the training data set S i The pseudo gain of i Select K unlabeled samples for manual labeling to obtain the training data set S i+1 , where the target detector M i Based on the training data set S i For the target detector M i-1 Semi-supervised training is performed, where i is the number of training rounds; after all unlabeled samples are labeled, the target detector M is obtained; The target detection module is used to perform target detection on the image to be detected based on the target detector M.

6. The platform according to claim 5, characterized in that The background logic also includes a rights management module, which is used for background user and user rights management logic.

7. The platform according to claim 5, characterized in that The interface of the training process management module includes: an instance creation class interface and a module control class interface; The instance creation class interface is used to create a semi-supervised training control module instance and an active learning sample screening module instance; The module control class interface is used for direct control of the active learning module and the semi-supervised learning module, and the communication control interface between the active learning module and the semi-supervised learning module.

8. The platform according to claim 7, characterized in that The module control class interface includes: a strategy modification class interface, a marking statistics and new addition class interface and a switching control class interface; The strategy modification interface is used to modify the active learning strategy and the semi-supervised training strategy; The annotation statistics and new class interface are used to analyze the training data set S i Statistics and annotation control interface of internal data; The switching control class interface is used to switch whether the current process is in active learning sample screening or in a semi-supervised training process.

9. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 4 when executed.

Images