A self-labeling method and system for instance object detection

Through clustering analysis and iterative training of self-labeling networks, a small number of labeled samples are used to perform self-labeling at the object detection level of instance objects, which solves the problem of low self-labeling quality in the prior art and realizes high-quality image self-labeling.

CN114092735BActive Publication Date: 2024-11-01BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010854935.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-24
Publication Date
2024-11-01
Estimated Expiration
2040-08-24

AI Technical Summary

Technical Problem

The existing image self-labeling method requires a large amount of manual annotation data at the object detection level of the instance object, and it is difficult to effectively use a small number of marked samples for self-labeling, resulting in low self-labeling quality.

Method used

By obtaining multiple pictures of the target object, performing color space conversion and clustering analysis, calculating information entropy for manual annotation, establishing a self-labeling network, using pseudo-labeling and score thresholds to update the annotation data, and gradually improving the accuracy and recall of the self-labeling network Rate.

Benefits of technology

It realizes high-quality self-labeling of unlabeled samples using a small number of marked samples, improves the accuracy and recall of self-labeling, overcomes the defects of relying on manual design features, and ensures the independence and collaborative training effect of the self-labeling network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114092735B_ABST
    Figure CN114092735B_ABST
Patent Text Reader

Abstract

The present invention relates to a self-annotation method and system for instance object target detection level, and relates to the field of image self-annotation. A self-annotation network is trained according to a set of manually annotated pictures, and a second self-annotation network is obtained by continuously training the first self-annotation network by updating the set of manually annotated pictures and the set of unannotated pictures. The self-annotation of unannotated pictures is realized by using the first self-annotation network and the second self-annotation network. The method and system provided by the present invention can realize self-annotation of unannotated instance samples by using a small amount of annotated instance samples, thereby improving the quality of self-annotation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image self-annotation, and in particular to a self-annotation method and system for instance object target detection level. Background Art

[0002] Current research on image self-annotation can be divided into two categories: based on traditional machine learning and based on deep learning.

[0003] The self-annotation method based on traditional machine learning mainly studies the category-level annotation of images. There are mainly self-annotation models based on statistical classification. This model classifies the semantic concept of each image as a category. Automatic image annotation can be converted into a multi-classification problem of images. Representative methods include support vector machines, two-dimensional multi-resolution Markov models, Gaussian mixture models (GMM), etc.; probability-based self-annotation models, which try to infer the correlation or joint probability distribution between images and semantic concepts. Representative methods include the Didycole allocation model, the consistent LDA model, the multi-Bernoulli correlation model (MBRM), etc.; self-annotation models based on the nearest neighbor model, which assumes that similar images have a high probability of sharing the same label. The representative method is the JEC model.

[0004] Self-labeling methods based on deep learning can automatically extract features from data and no longer rely on manually designed features. At present, self-labeling methods based on deep learning mainly study the text description level labeling of images, that is, to obtain the scene description of an image, such as the labeling result of "baseball player throwing a baseball". The mainstream methods are summarized as five methods based on multimodal space, multi-region, encoding and decoding, reinforcement learning and generative adversarial network. EasyDL realizes the self-labeling of the target detection level of images. Specifically, a part of the image data is manually selected and labeled to train the initial model, and then the trained model is used to predict the remaining unlabeled data. Then, through active learning, data with difficult categories to distinguish are selected, and then these data are manually labeled and added to the training set to train the model again. In addition to manually labeling the initial data, EasyDL also requires manual labeling of difficult-to-label data during the training process. The entire process is semi-automatic. Summary of the invention

[0005] The purpose of the present invention is to provide a self-labeling method and system for instance object target detection level, so as to realize self-labeling of unlabeled instance samples using a small amount of labeled instance samples, thereby improving the quality of self-labeling.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A self-labeling method for instance object detection level, comprising:

[0008] Get multiple pictures of the target object;

[0009] Performing color space conversion on multiple images of the target object to obtain one-dimensional color features, and performing cluster analysis based on the color features to obtain multiple clusters; each cluster includes multiple images;

[0010] Calculate the information entropy of each picture in each cluster, manually annotate the information entropy pictures within the set threshold range in each cluster to obtain a manually annotated picture set, obtain pictures that are not manually annotated in all clusters, and obtain the unannotated picture set;

[0011] Training a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network;

[0012] Self-annotating the unannotated picture set to obtain an updated manually annotated picture set and an unannotated picture set;

[0013] Using the updated manually annotated picture set to train the first self-annotation network to obtain a second self-annotation network;

[0014] Determine whether the number of pictures in the unlabeled picture set is not less than a set threshold, and obtain a first determination result;

[0015] If the first judgment result indicates that the number of pictures in the unlabeled picture set is not less than a set threshold, returning to the unlabeled picture set for self-labeling to obtain an updated manually labeled picture set and an unlabeled picture set;

[0016] If the first judgment result indicates that the number of pictures in the unlabeled picture set is less than a set threshold, then a manually labeled picture set is output.

[0017] Optionally, the unlabeled picture set is self-labeled to obtain an updated manually labeled picture set and an unlabeled picture set, which specifically includes:

[0018] Inputting the i-th picture in the unlabeled picture set into the first self-labeling network to obtain a first pseudo label of the i-th picture and a second pseudo label of the i-th picture;

[0019] Calculate the score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image;

[0020] Obtaining the score threshold of the cluster to which the i-th picture belongs;

[0021] Determine whether the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, and obtain a second determination result;

[0022] If the second judgment result indicates that the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, then the i-th picture, the first pseudo-label of the i-th picture, and the second pseudo-label of the i-th picture are added to the manually annotated picture set to update the manually annotated picture set; the i-th picture in the unannotated picture set is deleted to update the unannotated picture set;

[0023] If the second judgment result indicates that the score of the i-th picture is not greater than the score threshold of the cluster to which the i-th picture belongs, remove the first pseudo-label and the second pseudo-label of the i-th picture, and retain the i-th picture in the unlabeled picture set.

[0024] The above steps are performed on each picture in the unlabeled picture set to obtain a final updated manually labeled picture set and an unlabeled picture set.

[0025] Optionally, the step of training a self-annotation network according to a set of manually annotated images to obtain a first self-annotation network specifically includes:

[0026] A first self-labeling network is constructed according to a basic target detection network; the first self-labeling network includes a first target detection head module and a second target detection head module, and the first target detection head module and the second target detection head module both include a Classifier module and a Regressor module; the Classifier module and the Regressor module of the first target detection head module are respectively connected to the first key feature; the Classifier module and the Regressor module of the second target detection head module are respectively connected to the second key feature;

[0027] Using the manually annotated picture set as a training set to train the first self-annotation network;

[0028] Calculating a total loss function value of the self-labeling network; the total loss function value includes a total detection loss function value and a view difference loss function value;

[0029] According to the total loss function value of the self-labeling network, the weight of the first self-labeling network is updated using a gradient descent algorithm, and the step of training the first self-labeling network using the manually labeled image set as a training set is returned.

[0030] Optionally, the calculating the total loss function value of the self-labeling network specifically includes:

[0031] The first detection loss function value is calculated according to the formula loss_1=loss_cls_1+loss_regr_1; the first detection loss function value is the detection loss function value of the first target detection head module; wherein loss_cls_1 is the classification loss function value of the first target detection head module, loss_regr_1 is the regression loss function value of the first target detection head module, and loss_1 is the first detection loss function value;

[0032] The second detection loss function value is calculated according to the formula loss_2=loss_cls_2+loss_regr_2; the second detection loss function value is the detection loss function value of the second target detection head module; wherein loss_cls_2 is the classification loss function value of the second target detection head module, loss_regr_2 is the regression loss function value of the second target detection head module, and loss_2 is the second detection loss function value;

[0033] The total detection loss function value is calculated according to the formula loss_detection=loss_1+loss_2; ​​wherein loss_detection is the total detection loss function value;

[0034] According to the formula Calculate the first view function value; the first view function value is the view function value of the first target detection head module; wherein Gram1(i,j) is the first view function value, Feature1 is the first key feature, the first key feature is the three-dimensional feature of M*N*C, Feature1(m,n,i) is the feature value of the first key feature at (m,n,i), and Feature1(m,n,j) is the feature value of the first key feature at (m,n,j);

[0035] According to the formula Calculate the second view function value; the second view function value is the view function value of the second target detection head module; wherein Gram2(i,j) is the second view function value, Feature2 is the second key feature, the second key feature is the three-dimensional feature of M*N*C, Feature2(m,n,i) is the feature value of the second key feature at (m,n,i), and Feature2(m,n,j) is the feature value of the second key feature at (m,n,j);

[0036] The view difference loss function value is calculated according to the formula loss_gram=1 / MSE(Gram1(i,j),Gram2(i,j)); wherein loss_gram is the view difference loss function value, and MSE(Gram1(i,j),Gram2(i,j)) is the inverse of the mean square error of the first view function value and the second view function value;

[0037] The total loss function value is calculated according to the formula loss=loss_detection+loss_gram*q, where loss is the total loss function value and q is the balance parameter.

[0038] Optionally, obtaining the score threshold of the cluster to which the i-th picture belongs specifically includes:

[0039] Obtain the scores corresponding to all images in the cluster to which the i-th image belongs;

[0040] According to the formula Calculate the score threshold of the cluster; where t k is the score threshold of the kth cluster, is the score of the i-th image in the k-th cluster, N k is the number of images in the kth cluster, and β is the adjustment coefficient, which is used to adjust the percentage of self-annotated images to unannotated images each time.

[0041] A self-labeling system for instance object detection level, comprising:

[0042] An image acquisition module is used to acquire multiple images of the target object;

[0043] A cluster analysis module, used for performing cluster analysis on the multiple images of the target object to obtain multiple clusters; each cluster includes multiple images;

[0044] A manual labeling module is used to manually label the pictures in each cluster to obtain a manually labeled picture set and an unlabeled picture set; the manually labeled picture set is a set of pictures that are manually labeled in all clusters, and the unlabeled picture set is a set of pictures that are not manually labeled in all clusters;

[0045] A first self-annotation network construction module, used to train a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network;

[0046] a pseudo-label acquisition module, configured to input the i-th image in the unlabeled image set into the first self-labeling network to obtain a first pseudo-label for the i-th image and a second pseudo-label for the i-th image;

[0047] A score calculation module, used to calculate the score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image;

[0048] A score threshold calculation module, used to obtain the score threshold of the cluster to which the i-th picture belongs;

[0049] a second judgment module, configured to judge whether the score of the i-th picture is greater than a score threshold of the cluster to which the i-th picture belongs, and obtain a second judgment result;

[0050] a set updating module, configured to, when the second judgment result indicates that the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, add the i-th picture, the first pseudo-label of the i-th picture, and the second pseudo-label of the i-th picture to the manually annotated picture set to update the manually annotated picture set; and delete the i-th picture in the unannotated picture set to update the unannotated picture set;

[0051] a set retention module, configured to, when the second judgment result indicates that the score of the ith image is not greater than the score threshold of the cluster to which the ith image belongs, remove the first pseudo label of the ith image and the second pseudo label of the ith image, retain the ith image in the unlabeled image set, use the (i+1)th image in the unlabeled image set as the ith image, and return to the pseudo label acquisition module;

[0052] The second self-labeling network construction module is used to train the first self-labeling network using the updated set of manually labeled pictures to obtain a second self-labeling network.

[0053] The first judgment module is used to judge whether the number of pictures in the unlabeled picture set is not less than a set threshold to obtain a first judgment result.

[0054] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0055] The present invention provides a self-labeling method and system for instance object target detection level, wherein a self-labeling network is trained through a small amount of labeled images, and then the trained self-labeling network is used to predict unlabeled images, and a set of manually labeled images is updated according to the prediction results, and then the prediction is continued according to the updated manually labeled images and the trained self-labeling network, the quality of self-labeling is ensured by comprehensively considering the accuracy and recall rate, and the unlabeled images are self-labeled through a small amount of labeled images, thereby realizing the self-labeling of images.

[0056] In addition, by using the total loss function to train the self-annotation network, the extracted image features are decorrelated in the collaborative training, achieving view independence of the collaborative training. Moreover, by establishing a self-annotation network to automatically extract image features, the defect of relying on the designer's experience is overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 A flowchart of a self-labeling method for object target detection level of an example of the present invention;

[0059] Figure 2 A schematic diagram of target detection level annotation in the self-annotation method for target detection level of an example of the present invention;

[0060] Figure 3 A schematic diagram of the structure of a self-annotation network in a self-annotation method for object detection level of an example of the present invention;

[0061] Figure 4 A schematic diagram of the data processing flow in the self-labeling method for object target detection level of the present invention;

[0062] Figure 5 Schematic diagram of prediction results of the self-labeling method for object detection level of the present invention;

[0063] Figure 6 A schematic diagram of the structure of a self-labeling system at the object detection level of an example of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The purpose of the present invention is to provide a self-labeling method and system for instance object target detection level, so as to realize self-labeling of unlabeled instance samples using a small amount of labeled instance samples, thereby improving the quality of self-labeling.

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Embodiment 1

[0068] like Figure 1 As shown, this embodiment provides a self-labeling method for an example object target detection level, including:

[0069] A self-labeling method for instance object detection level, comprising:

[0070] Step 101: Acquire multiple images of the target object;

[0071] Step 102: performing color space conversion on the multiple images of the target object to obtain one-dimensional color features, and performing cluster analysis based on the color features to obtain multiple clusters; each cluster includes multiple images;

[0072] Step 103: Calculate the information entropy of each picture in each cluster, manually annotate the information entropy pictures within the set threshold range in each cluster to obtain a manually annotated picture set, obtain pictures that are not manually annotated in all clusters, and obtain the unannotated picture set;

[0073] Step 104: training a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network;

[0074] Step 105: self-annotate the unannotated picture set to obtain an updated manually annotated picture set and an unannotated picture set;

[0075] Step 106: training the first self-annotation network using the updated manually annotated picture set to obtain a second self-annotation network;

[0076] Step 107: Determine whether the number of pictures in the unlabeled picture set is not less than a set threshold, and obtain a first judgment result; if the first judgment result indicates that the number of pictures in the unlabeled picture set is not less than the set threshold, return to step 105; if the first judgment result indicates that the number of pictures in the unlabeled picture set is less than the set threshold, execute step 108.

[0077] Step 108: Output the manually annotated image set.

[0078] Step 104 specifically includes:

[0079] A first self-labeling network is constructed according to a basic target detection network that meets the accuracy requirements; wherein the basic target detection network in this embodiment is a RetinaNet detection network; the first self-labeling network includes a first target detection head module and a second target detection head module, and the first target detection head module and the second target detection head module both include a Classifier module and a Regressor module; the Classifier module and the Regressor module of the first target detection head module are respectively connected to the first key feature; the Classifier module and the Regressor module of the second target detection head module are respectively connected to the second key feature;

[0080] Using the manually annotated picture set as a training set to train the first self-annotation network;

[0081] Calculating a total loss function value of the self-labeling network; the total loss function value includes a total detection loss function value and a view difference loss function value;

[0082] According to the total loss function value of the self-labeling network, the weight of the first self-labeling network is updated using a gradient descent algorithm, and the step of training the first self-labeling network using the manually labeled image set as a training set is returned.

[0083] The calculating of the total loss function value of the self-labeling network specifically includes:

[0084] The first detection loss function value is calculated according to the formula loss_1=loss_cls_1+loss_regr_1; the first detection loss function value is the detection loss function value of the first target detection head module; wherein loss_cls_1 is the classification loss function value of the first target detection head module, loss_regr_1 is the regression loss function value of the first target detection head module, and loss_1 is the first detection loss function value;

[0085] The second detection loss function value is calculated according to the formula loss_2=loss_cls_2+loss_regr_2; the second detection loss function value is the detection loss function value of the second target detection head module; wherein loss_cls_2 is the classification loss function value of the second target detection head module, loss_regr_2 is the regression loss function value of the second target detection head module, and loss_2 is the second detection loss function value;

[0086] The total detection loss function value is calculated according to the formula loss_detection=loss_1+loss_2; ​​wherein loss_detection is the total detection loss function value;

[0087] According to the formula Calculate the first view function value; the first view function value is the view function value of the first target detection head module; wherein Gram1(i,j) is the first view function value, Feature1 is the first key feature, the first key feature is the three-dimensional feature of M*N*C, Feature1(m,n,i) is the feature value of the first key feature at (m,n,i), and Feature1(m,n,j) is the feature value of the first key feature at (m,n,j);

[0088] According to the formula Calculate the second view function value; the second view function value is the view function value of the second target detection head module; wherein Gram2(i,j) is the second view function value, Feature2 is the second key feature, the second key feature is the three-dimensional feature of M*N*C, Feature2(m,n,i) is the feature value of the second key feature at (m,n,i), and Feature2(m,n,j) is the feature value of the second key feature at (m,n,j);

[0089] The view difference loss function value is calculated according to the formula loss_gram=1 / MSE(Gram1(i,j),Gram2(i,j)); wherein loss_gram is the view difference loss function value, and MSE(Gram1(i,j),Gram2(i,j)) is the inverse of the mean square error of the first view function value and the second view function value;

[0090] The total loss function value is calculated according to the formula loss=loss_detection+loss_gram*q, where loss is the total loss function value and q is the balance parameter.

[0091] Wherein, step 105 specifically includes:

[0092] Inputting the i-th picture in the unlabeled picture set into the first self-labeling network to obtain a first pseudo label of the i-th picture and a second pseudo label of the i-th picture;

[0093] Calculate the score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image;

[0094] Obtaining the score threshold of the cluster to which the i-th picture belongs;

[0095] Determine whether the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, and obtain a second determination result;

[0096] If the second judgment result indicates that the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, then the i-th picture, the first pseudo-label of the i-th picture, and the second pseudo-label of the i-th picture are added to the manually annotated picture set to update the manually annotated picture set; the i-th picture in the unannotated picture set is deleted to update the unannotated picture set;

[0097] If the second judgment result indicates that the score of the i-th picture is not greater than the score threshold of the cluster to which the i-th picture belongs, remove the first pseudo-label and the second pseudo-label of the i-th picture, and retain the i-th picture in the unlabeled picture set.

[0098] The above steps are performed on each picture in the unlabeled picture set to obtain a final updated manually labeled picture set and an unlabeled picture set.

[0099] The step of obtaining the score threshold of the cluster to which the i-th picture belongs specifically includes:

[0100] Obtain the scores corresponding to all images in the cluster to which the i-th image belongs;

[0101] According to the formula Calculate the score threshold of the cluster; where t k is the score threshold of the kth cluster, is the score of the i-th image in the k-th cluster, N k is the number of images in the kth cluster, and β is the adjustment coefficient, which is used to adjust the percentage of self-annotated images to unannotated images each time.

[0102] Embodiment 2

[0103] This embodiment provides a specific application scenario of a self-labeling method for an instance object target detection level. Figure 4 In the home scene, the goal is to self-label a large number of common household instance objects. The target objects in this embodiment include three instance objects: Red Bull bottle, TV, and mineral water bottle. The meaning of labeling is to label the frame of the target object (target detection level). Figure 2 shown.

[0104] Step 1: Data collection. The kitchen area is photographed to obtain M manually taken pictures of the target objects (Red Bull bottles, TVs, and mineral water bottles) in the kitchen scene.

[0105] Step 2: Label the collected images. Manually label a small number of the collected images, and finally obtain A manually labeled images and B unlabeled images. The specific process is as follows:

[0106] S1: Calculate the color features of M manually taken pictures. For each picture, convert the image from RGB to HSV color space. Count the color histograms of the three components H, S, and V respectively, with the H component bins set to 17, and the S and V bins set to 4. Then convert the three-dimensional histogram into a one-dimensional feature of 17*4*4(272) as the feature of the picture, thus obtaining M pictures and a one-dimensional color feature of length 272 corresponding to each picture.

[0107] S2: Based on color features, k-means clustering is performed on the collected images to obtain multiple clusters. Specifically, the following steps are included:

[0108] S2-1: Define the number of clusters as K, randomly select K images from M images as centroids, and each centroid determines a cluster. The centroid corresponding to the i-th image is ci, and the corresponding cluster is si (i = 1, 2, 3, ... k).

[0109] S2-2: Using the formula Calculate the distance between image x and each centroid. Where dist(ci,x) represents the distance between image x and centroid ci, ci_fea[i], x_fea[i] are the one-dimensional color features of image ci and image x with a length of 272 respectively.

[0110] According to the formula Assign the image x to the cluster si corresponding to the centroid ci. Assign the M images in sequence.

[0111] S2-3: According to the formula Recalculate K centroids. Among them, c i is the centroid corresponding to the i-th image, x i is the i-th image x, and si is the i-th cluster.

[0112] S2-4: Repeat S2-2 and S2-3, and reallocate the M images to K clusters until the images in the K clusters remain unchanged. No change means that after re-executing step S2-2, the cluster number to which each image belongs remains unchanged. i ∈s j (i=1,2,......M;j=1,2,......K) does not change for any i and j. For example, after the last execution of step S2-2, picture 1 belongs to cluster 1 (expressed as x1∈s1), and picture 2 belongs to cluster 5 (expressed as x2∈s5). After the execution of step S2-2 this time, picture 1 still belongs to cluster 1 (expressed as x1∈s1), and picture 2 still belongs to cluster 5 (expressed as x2∈s5). For M pictures, if the above conditions are met, then the pictures in K clusters are said to remain unchanged.

[0113] S2-5: According to the formula Calculate the silhouette coefficient of each image (M images, calculated M times), where s(i) is the silhouette coefficient of the i-th image, a(i) is the average distance from the i-th image to other images in its cluster, and b(i) is the average distance from the i-th image to all images in other clusters. The total silhouette coefficient R is the average of the silhouette coefficients of the M images.

[0114] S2-6: Select different cluster numbers K and repeat steps (S2-1 to S2-5) to obtain the silhouette coefficients R corresponding to different cluster numbers K. Select the cluster number K when the silhouette coefficient R is the largest as the final cluster number, and execute steps (S2-1 to S2-4) again to finally obtain K clusters, each of which contains N k The total number of pictures is M.

[0115] S3: For K clusters, each cluster contains N k The information entropy is calculated for each picture taken, and the pictures are manually annotated based on the information entropy.

[0116] According to the formula Calculate the information entropy E of each picture, where p j is the proportion of pixels with grayscale value j in the image. For K clusters, select W images with the highest information entropy E value in each cluster for manual annotation. So far, A manually annotated images and B unannotated images are obtained, where A = W*K and B = MW*K. A manually annotated image has annotation information, and each image corresponds to an XML file that stores the location of the target object in the image. B unannotated images only have images, no annotation information, and only corresponding XML files.

[0117] Step 3: Build a self-labeling network based on the RetinaNet detection network and train the constructed self-labeling network. Figure 3 As shown in the figure, A manually annotated pictures constitute the manually annotated picture set, and B unannotated pictures constitute the unannotated picture set. The specific process is as follows:

[0118] S1: Build a self-annotation network based on the RetinaNet detection network.

[0119] The object detection network based on deep learning consists of three parts: feature extraction module Backbone, region of interest acquisition module RoiExtractor and object detection module RoiHead; the feature extraction module extracts features from the original image, the region of interest acquisition module extracts features of the region of interest from the feature map, and the object detection module uses the features of the region of interest for classification and regression. The Backbone and RoiExtractor of RetinaNet are retained, and the single RoiHead of RetinaNet is expanded into two, namely the first object detection head module RoiHead1 and the second object detection head module RoiHead2. The first object detection head module RoiHead1 and the second object detection head module RoiHead2 share Backbone and RoiExtractor, and the first object detection head module RoiHead1 and the second object detection head module RoiHead2 both include Classifier module and Regressor module, the Classifier module is used for category judgment, and the Regressor module is used for bounding box regression. The weight θ0 of the self-annotation network is randomly initialized. A self-annotation network with random initial weight θ0 is obtained.

[0120] S2: Dataset settings.

[0121] Two training sets φ1 and φ2 are set to store images with labeled information, that is, training sets φ1 and φ2 are both manually labeled image sets, φ1 is used to calculate the lost label information during the back propagation of the first target detection head module RoiHead1, and obtain the pseudo label A1, φ2 is used to calculate the lost label information during the back propagation of the second target detection head module RoiHead2, and obtain the pseudo label A2. An unlabeled image pool φ3 is set to store unlabeled images, and the unlabeled image pool is a collection of unlabeled images.

[0122] S3: Train the self-labeling network. Back-propagate the images in the training set φ1, φ2 through the self-labeling network with random initial weights θ0. The process is as follows:

[0123] S3-1: Calculate the total loss function value. The specific process is as follows:

[0124] S3-1-1: Calculate the detection loss of the target detection module.

[0125] The classification loss function value loss_cls_1 and the regression loss function value loss_regr_1 of the first object detection head module RoiHead1 are obtained by using the labeled information in the training set φ1. Therefore, the detection loss of the first object detection head module RoiHead1 is calculated as: loss_1 = loss_cls_1 + loss_regr_1.

[0126] The classification loss function value loss_cls_2 and the regression loss function value loss_regr_2 of the second object detection head module RoiHead2 are obtained by using the labeled information in the training set φ2. Therefore, the detection loss of the second object detection head module RoiHead2 is calculated as: loss_2 = loss_cls_2 + loss_regr_2.

[0127] Therefore, the total detection loss function value loss_detection is: loss_detection=loss_1+loss_2.

[0128] S3-1-2: Calculate the loss_gram view difference loss function value.

[0129] The example of the present invention is a self-labeling network built with RetinaNet as the detection network, so the first layer feature feature1 of the RoiHead1 module is the output of the convolution layer, which is a three-dimensional tensor with a dimension of (M, N, C). In addition, a basic detection network with a detection accuracy of not less than 70% on the public database VOC2007 dataset, such as FasterR-CNN, R-FCN, SSD, etc. can be used as the detection network.

[0130] According to the formula Calculate the first view function value; the first view function value is the view function value of the first target detection head module; wherein, Gram1(i,j) is the first view function value, Feature1 is the first key feature, the first key feature is the three-dimensional feature of M*N*C, Feature1(m,n,i) is the feature value of the first key feature at (m,n,i), and Feature1(m,n,j) is the feature value of the first key feature at (m,n,j).

[0131] According to the formula Calculate the second view function value; the second view function value is the view function value of the second target detection head module; wherein, Gram2(i,j) is the second view function value, Feature2 is the second key feature, the second key feature is the three-dimensional feature of M*N*C, Feature2(m,n,i) is the feature value of the second key feature at (m,n,i), and Feature2(m,n,j) is the feature value of the second key feature at (m,n,j).

[0132] The view difference loss function value is calculated according to the formula loss_gram=1 / MSE(Gram1(i,j),Gram2(i,j)); where loss_gram is the view difference loss function value, and MSE(Gram1(i,j),Gram2(i,j)) is the inverse of the mean square error between the first view function value and the second view function value.

[0133] S3-1-3: Calculate the total loss function value loss.

[0134] The total loss function value loss consists of two parts: the total detection loss function value loss_detection and the view difference loss function value loss_gram. The total loss function value is calculated according to the formula loss = loss_detection + loss_gram*q, where loss is the total loss function value and q is the balance parameter.

[0135] S3-2: Update self-labeling network weights.

[0136] For the weight θ in the self-labeling network, set the learning rate σ and use the gradient descent algorithm θ1=θ-σ▽ θ loss updates the random initial weight θ0 of the self-labeled network, where ▽ θ Loss is the gradient of the total loss function value loss with respect to the weight θ. Iterate the training until the loss converges without obvious decrease, and stop the training. At this point, a self-labeled network with initial weight θ1 is obtained.

[0137] Step 4: Self-annotate the unannotated images.

[0138] S1: forward propagation and calculation of scores.

[0139] The unlabeled image x in the unlabeled image pool φ3 is forward propagated in the self-labeling network with initial weight θ1 to obtain the prediction results of the first target detection head module RoiHead1 and the second target detection head module RoiHead2, namely pseudo labels A1 and A2. The pseudo label contains the prediction box in the image and the classification confidence w of the prediction box. i is the target object category that exists in both pseudo labels A1 and A2. accuracy1(w i ), accuracy2(w i ) are the classification confidences of A1 and A2 for the category respectively. The number of categories with a confidence threshold greater than δ is added together to get the score score(x) of the image x. According to the formula Calculate the score score(x) of image x. Output the score score(x) and pseudo-labels A1, A2 of each image in the unlabeled image pool φ3.

[0140] Specific as Figure 5 As shown, the first target detection head module has three prediction boxes for the image:

[0141] Prediction box 1: Red Bull with a classification confidence of 0.78

[0142] Prediction box 2: TV with a classification confidence of 0.95

[0143] Prediction box 3: mineral water bottle with classification confidence of 0.65

[0144] The second target detection head module has two prediction boxes for the image:

[0145] Prediction box 4: Red Bull with a classification confidence of 0.85

[0146] Prediction box 5: TV with a classification confidence of 0.86

[0147] Then the pseudo label A1 is predicted box 1, 2, 3, and the pseudo label A2 is predicted box 4, 5.

[0148] w i are the target object categories that exist in both pseudo-labels A1 and A2, namely Red Bull and TV.

[0149] According to the formula Calculate the score for the image.

[0150] n = 2 (Red Bull and TV), assuming the confidence threshold δ is 0.8

[0151] For Red Bull, accuracy1(w i )=0.78,accuracy2(w i )=0.85, the condition is not met and Red Bull is not a valid category.

[0152] For TV, accuracy1(w i )=0.95,accuracy2(w i )=0.86, the condition is met, the TV is a valid category, and the score is increased by 1.

[0153] Therefore the image has a score of 1.

[0154] If Red Bull If true, the score is increased by 1. If the television is a valid category, the score of the picture is 2.

[0155] S2: Set the score threshold.

[0156] Each image in the unlabeled image pool φ3 belongs to a cluster. For the kth cluster, it contains N kpictures, and set the score threshold t of the cluster according to the score of each picture in the cluster k According to the formula Calculate the score threshold t k .in, represents the i-th image in the k-th cluster, For pictures , β is the adjustment coefficient, which is used to adjust the percentage of self-annotated pictures to unannotated pictures each time.

[0157] S3: For each image x (belonging to cluster K) in the unlabeled image pool φ3, determine whether the score score(x) of image x is greater than the cluster score threshold t k .

[0158] If there is an image x that satisfies the above conditions, output the image x and its pseudo labels A1, A2. Then execute step 5.

[0159] If the above conditions are not met, the current image x is added to the training set as a self-annotated image. At this time, the training sets φ1 and φ2 include A manually annotated images and self-annotated images. The image in the unannotated image set is deleted. The next image in the unannotated image set is self-annotated.

[0160] Step 5: Retrain.

[0161] S1: Update the dataset.

[0162] Delete the image from the unlabeled image pool φ3 to obtain an updated unlabeled image pool φ3'.

[0163] Add the image with pseudo label to φ1, φ2 to get updated training sets φ1', φ2'.

[0164] S2: Retraining.

[0165] The updated training set φ1', φ2' is used to retrain the self-annotation network. The training process is exactly the same as the S3 operation in step 2. Finally, an updated self-annotation network with a weight of θ2 and an unannotated image pool φ3' are obtained.

[0166] In this embodiment, for each unlabeled image, steps 4 and 5 are repeated to iteratively perform self-labeling until no unlabeled image with a score greater than the score threshold can be obtained in step 3, and the process is stopped to obtain all self-labeled images.

[0167] As another embodiment, when constructing a self-annotation network, the present invention can also construct a self-annotation network using FasterR-CNN as the detection network. In this case, the first key feature D1 of the first target detection head module is a one-dimensional vector with a dimension of C. In this embodiment, the first view function value is calculated according to the formula Gram(i,j)=Feature(i)*Feature(j), Gram1(i,j) is the first view function value, Feature(i) is the feature value of the first key feature D1 at (i); Feature(j) is the feature value of the first key feature D1 at (j).

[0168] The second view function value is calculated according to the formula Gram(i,j)=Feature(i)*Feature(j), where Gram2(i,j) is the first view function value, Feature(i) is the feature value of the second key feature D2 at (i), and Feature(j) is the feature value of the second key feature D2 at (j).

[0169] The view difference loss function value is calculated according to the formula loss_gram=1 / MSE(Gram1(i,j),Gram2(i,j)), where loss_gram is the view difference loss function value and MSE(Gram1(i,j),Gram2(i,j)) is the inverse of the mean square error between the first view function value and the second view function value.

[0170] Embodiment 3

[0171] like Figure 6 As shown, the present invention provides a self-labeling system for instance object target detection level, including a self-labeling system for instance object target detection level, including:

[0172] The picture acquisition module 601 is used to acquire multiple pictures of the target object;

[0173] A cluster analysis module 602 is used to perform cluster analysis on the multiple images of the target object to obtain multiple clusters; each cluster includes multiple images;

[0174] The manual labeling module 603 is used to manually label the pictures in each cluster to obtain a manually labeled picture set and an unlabeled picture set; the manually labeled picture set is a set of pictures that are manually labeled in all clusters, and the unlabeled picture set is a set of pictures that are not manually labeled in all clusters;

[0175] A first self-annotation network construction module 604 is used to train a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network;

[0176] A pseudo-label acquisition module 605 is used to input the i-th picture in the unlabeled picture set into the first self-labeling network to obtain a first pseudo-label of the i-th picture and a second pseudo-label of the i-th picture;

[0177] A score calculation module 606, configured to calculate a score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image;

[0178] A score threshold calculation module 607 is used to obtain a score threshold of the cluster to which the i-th picture belongs;

[0179] A second judgment module 608 is used to judge whether the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, and obtain a second judgment result;

[0180] A set updating module 609 is configured to, when the second judgment result indicates that the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, add the i-th picture, the first pseudo-label of the i-th picture, and the second pseudo-label of the i-th picture to the manually annotated picture set to update the manually annotated picture set; and delete the i-th picture in the unannotated picture set to update the unannotated picture set;

[0181] A set retaining module 610 is configured to, when the second judgment result indicates that the score of the i-th picture is not greater than the score threshold of the cluster to which the i-th picture belongs, remove the first pseudo label of the i-th picture and the second pseudo label of the i-th picture, retain the i-th picture in the unlabeled picture set, use the i+1th picture in the unlabeled picture set as the i-th picture, and return to the pseudo label acquisition module;

[0182] The second self-annotation network construction module 611 is used to train the first self-annotation network using the updated set of manually annotated pictures to obtain a second self-annotation network.

[0183] The first judgment module 612 is used to judge whether the number of pictures in the unlabeled picture set is not less than a set threshold to obtain a first judgment result.

[0184] Advantages of the self-labeling method and system for instance object detection level provided by the present invention:

[0185] 1. Build a self-annotation network based on convolutional neural network to automatically extract image features.

[0186] 2. During data collection, key samples are selected for manual labeling to ensure that the self-labeling network has good generalization ability during initial training. The self-labeling strategy is designed to comprehensively consider accuracy and recall rate to ensure the quality of self-labeling.

[0187] 3. The loss-gram loss function is proposed to decorrelate the extracted image features during the collaborative training process, meeting the view independence requirements of collaborative training.

[0188] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0189] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A self-labeling method for instance object detection level, characterized in that: include: Get multiple pictures of the target object; Performing color space conversion on multiple images of the target object to obtain one-dimensional color features, and performing cluster analysis based on the color features to obtain multiple clusters; each cluster includes multiple images; Calculate the information entropy of each image in each cluster, manually annotate the images in each cluster whose information entropy is within the set threshold range to obtain a set of manually annotated images, obtain the images in all clusters that have not been manually annotated to obtain a set of unannotated images; Training a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network; Self-annotating the unannotated picture set to obtain an updated manually annotated picture set and an unannotated picture set; Using the updated manually annotated picture set to train the first self-annotation network to obtain a second self-annotation network; Determine whether the number of pictures in the unlabeled picture set is not less than a set threshold, and obtain a first determination result; If the first judgment result indicates that the number of pictures in the unlabeled picture set is not less than the set threshold, returning to the step of self-labeling the unlabeled picture set to obtain an updated manually labeled picture set and an unlabeled picture set; If the first judgment result indicates that the number of pictures in the unlabeled picture set is less than a set threshold, then a manually labeled picture set is output.

2. The method for self-labeling of instance object target detection level according to claim 1, characterized in that: The self-annotating the unannotated picture set to obtain an updated manually annotated picture set and an unannotated picture set specifically includes: Inputting the i-th picture in the unlabeled picture set into the first self-labeling network to obtain a first pseudo label of the i-th picture and a second pseudo label of the i-th picture; Calculate the score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image; Obtaining the score threshold of the cluster to which the i-th picture belongs; Determine whether the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, and obtain a second determination result; If the second judgment result indicates that the score of the i-th picture is greater than the score threshold of the cluster to which the i-th picture belongs, then the i-th picture, the first pseudo-label of the i-th picture, and the second pseudo-label of the i-th picture are added to the manually annotated picture set to update the manually annotated picture set; the i-th picture in the unannotated picture set is deleted to update the unannotated picture set; If the second judgment result indicates that the score of the i-th picture is not greater than the score threshold of the cluster to which the i-th picture belongs, removing the first pseudo-label and the second pseudo-label of the i-th picture, and retaining the i-th picture in the unlabeled picture set; The above steps are performed on each picture in the unlabeled picture set to obtain a final updated manually labeled picture set and an unlabeled picture set.

3. The method for self-labeling of instance object target detection level according to claim 1, characterized in that: The step of training the self-annotation network according to the manually annotated picture set to obtain the first self-annotation network specifically includes: A first self-labeling network is constructed according to a basic target detection network; the first self-labeling network includes a first target detection head module and a second target detection head module, and the first target detection head module and the second target detection head module both include a Classifier module and a Regressor module; the Classifier module and the Regressor module of the first target detection head module are respectively connected to the first key feature; the Classifier module and the Regressor module of the second target detection head module are respectively connected to the second key feature; Using the manually annotated picture set as a training set to train the first self-annotation network; Calculating a total loss function value of the self-labeling network; the total loss function value includes a total detection loss function value and a view difference loss function value; According to the total loss function value of the self-labeling network, the weight of the first self-labeling network is updated using a gradient descent algorithm, and the step of training the first self-labeling network using the manually labeled image set as a training set is returned.

4. The method for self-labeling of instance object target detection level according to claim 3, characterized in that: The calculating of the total loss function value of the self-labeling network specifically includes: The first detection loss function value is calculated according to the formula loss_1=loss_cls_1+loss_regr_1; the first detection loss function value is the detection loss function value of the first target detection head module; wherein loss_cls_1 is the classification loss function value of the first target detection head module, loss_regr_1 is the regression loss function value of the first target detection head module, and loss_1 is the first detection loss function value; The second detection loss function value is calculated according to the formula loss_2=loss_cls_2+loss_regr_2; the second detection loss function value is the detection loss function value of the second target detection head module; wherein loss_cls_2 is the classification loss function value of the second target detection head module, loss_regr_2 is the regression loss function value of the second target detection head module, and loss_2 is the second detection loss function value; The total detection loss function value is calculated according to the formula loss_detection=loss_1+loss_2; ​​wherein loss_detection is the total detection loss function value; According to the formula Calculate the first view function value; the first view function value is the view function value of the first target detection head module; wherein Gram1(i,j) is the first view function value, Feature1 is the first key feature, the first key feature is the three-dimensional feature of M*N*C, Feature1(m,n,i) is the feature value of the first key feature at (m,n,i), and Feature1(m,n,j) is the feature value of the first key feature at (m,n,j); According to the formula Calculate the second view function value; the second view function value is the view function value of the second target detection head module; wherein Gram2(i,j) is the second view function value, Feature2 is the second key feature, the second key feature is the three-dimensional feature of M*N*C, Feature2(m,n,i) is the feature value of the second key feature at (m,n,i), and Feature2(m,n,j) is the feature value of the second key feature at (m,n,j); The view difference loss function value is calculated according to the formula loss_gram=1 / MSE(Gram1(i,j),Gram2(i,j)); wherein loss_gram is the view difference loss function value, and MSE(Gram1(i,j),Gram2(i,j)) is the inverse of the mean square error of the first view function value and the second view function value; The total loss function value is calculated according to the formula loss=loss_detection+loss_gram*q, where loss is the total loss function value and q is the balance parameter.

5. The method for self-labeling of instance object target detection level according to claim 2, characterized in that: The step of obtaining the score threshold of the cluster to which the i-th picture belongs specifically includes: Obtain the scores corresponding to all images in the cluster to which the i-th image belongs; According to the formula Calculate the score threshold of the cluster; where t k is the score threshold of the kth cluster, is the score of the i-th image in the k-th cluster, N k is the number of images in the kth cluster, and β is the adjustment coefficient, which is used to adjust the percentage of self-annotated images to unannotated images each time.

6. A self-labeling system for instance object detection level, characterized in that: include: An image acquisition module is used to acquire multiple images of the target object; A cluster analysis module, used for performing cluster analysis on the multiple images of the target object to obtain multiple clusters; each cluster includes multiple images; A manual labeling module is used to manually label some images in each cluster to obtain a set of manually labeled images and a set of unlabeled images; The manually annotated picture set is a set of pictures that are manually annotated in all clusters, and the unannotated picture set is a set of pictures that are not manually annotated in all clusters; A first self-annotation network construction module, used to train a self-annotation network according to the manually annotated picture set to obtain a first self-annotation network; a pseudo-label acquisition module, configured to input the i-th image in the unlabeled image set into the first self-labeling network to obtain a first pseudo-label for the i-th image and a second pseudo-label for the i-th image; A score calculation module, used to calculate the score of the i-th image according to the first pseudo label and the second pseudo label of the i-th image; A score threshold calculation module, used to obtain the score threshold of the cluster to which the i-th picture belongs; a second judgment module, configured to judge whether the score of the i-th picture is greater than a score threshold of the cluster to which the i-th picture belongs, and obtain a second judgment result; a set updating module, configured to, when the second judgment result indicates that the score of the ith picture is greater than the score threshold of the cluster to which the ith picture belongs, add the ith picture, the first pseudo-label of the ith picture, and the second pseudo-label of the ith picture to the manually annotated picture set to update the manually annotated picture set; Deleting the i-th picture in the unlabeled picture set to update the unlabeled picture set; a set retention module, configured to, when the second judgment result indicates that the score of the ith image is not greater than the score threshold of the cluster to which the ith image belongs, remove the first pseudo label of the ith image and the second pseudo label of the ith image, retain the ith image in the unlabeled image set, use the (i+1)th image in the unlabeled image set as the ith image, and return to the pseudo label acquisition module; A second self-annotation network construction module, configured to train the first self-annotation network using the updated manually annotated picture set to obtain a second self-annotation network; The first judgment module is used to judge whether the number of pictures in the unlabeled picture set is not less than a set threshold to obtain a first judgment result.