A semi-supervised object detection method

Abstract

This application discloses a semi-supervised target detection method. First, multi-scale scaling data enhancement is added to the data set; then, a multi-scale regularization loss function is proposed for the region candidate network RPN and the region prediction head RoIHead of the two-stage target detection model FasterR-CNN, so that the model outputs the same prediction results for the image and the corresponding maximum pooling downsampled image, thereby improving the robustness of the model. In order to solve the problem that the semi-supervised target detection algorithm ignores the calculation of the positioning loss of unlabeled data, this application proposes a re-prediction algorithm to measure the positioning quality of pseudo-labels, and at the same time, the accurate and reliable classification pseudo-labels are used for the calculation of the positioning loss of unlabeled data to improve the positioning accuracy of the model.

Description

Technical Field

[0001] The present application belongs to the field of model detection technology, and specifically relates to a semi-supervised target detection method. Background Art

[0002] At present, semi-supervised object detection methods are generally divided into two categories: semi-supervised object detection methods based on consistency regularization and semi-supervised object detection methods based on pseudo labels. The former uses consistency regularization to constrain the model to output the same prediction results for perturbed and unperturbed images. The latter is a more mainstream semi-supervised object detection method, which uses the teacher model to predict the pseudo labels of unlabeled data; then, the labeled data and unlabeled data with strong data enhancement are used to train the student model; finally, the exponential moving average (EMA) is used to update the parameters of the teacher model according to the parameters of the student model.

[0003] Although the semi-supervised target detection method utilizes the information of unlabeled data, it cannot effectively solve the problem of missed detection of multi-scale targets in target detection. In addition, based on the target detection results of the teacher model, a part of high-quality reliable classification pseudo-labels can be screened out by using the threshold method based on the highest category score, but the category score can only reflect the classification of the target. Therefore, the label obtained by threshold screening is only used for the classification loss calculation of unlabeled data, and is not used for the localization loss calculation of unlabeled data. Summary of the invention

[0004] The present application provides a semi-supervised target detection method to solve the problem that the existing semi-supervised target detection method ignores the calculation of unlabeled data positioning loss.

[0005] In order to solve the above technical problems, a technical solution adopted in this application is: a semi-supervised target detection method, comprising:

[0006] The image dataset is divided into labeled data and unlabeled data, and data enhancement is performed on the labeled data and the unlabeled data; the amount of unlabeled data is greater than that of labeled data;

[0007] Based on the two-stage target detection algorithm FasterR-CNN, obtain the teacher model and student model;

[0008] Based on the teacher model, the unlabeled data is screened and the classification loss is calculated by the threshold method to obtain multi-scale regularization loss and reliable classification pseudo labels;

[0009] Based on the teacher model, the unlabeled data is screened through the re-prediction method and the bounding box regression loss is calculated to obtain the unsupervised positioning loss;

[0010] Obtain supervised loss based on the student model and labeled data;

[0011] Obtain the final model loss function based on multi-scale regularization loss, unsupervised localization loss, and supervised loss;

[0012] The student model updates the first parameter using gradient descent based on the final model loss function; the teacher model updates the second parameter using the EMA method based on the first parameter.

[0013] Furthermore, the method for data enhancement for labeled data and unlabeled data includes:

[0014] Multi-scale scaling is used for data enhancement on unlabeled and labeled data, randomly scaling the image width and height in equal proportions.

[0015] Furthermore, the range of proportional scaling is -50%-150%.

[0016] Furthermore, based on the teacher model, the unlabeled data is screened by the threshold method and the classification loss is calculated to obtain the multi-scale regularization loss and reliable classification pseudo-label method, including:

[0017] Based on inputting the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead of the teacher model, obtaining a first prediction result; wherein the first prediction result includes the frame coordinates and category score of the target candidate box;

[0018] Constrain the output results of the region candidate network RPN and the region prediction head RoIHead to obtain multi-scale regularization loss;

[0019] In response to a highest category score of the target candidate box being greater than a first threshold, a reliable classification pseudo label is obtained.

[0020] Furthermore, the output results of the region candidate network RPN and the region prediction head RoIHead are constrained to obtain a method for multi-scale regularization loss, including:

[0021] Perform maximum pooling downsampling on the first image in the unlabeled data to obtain a first sampled image;

[0022] Input the first image and the first sampled image into the backbone network of the teacher model and extract features, and use the region proposal network RPN to generate a first candidate box and obtain a first loss function;

[0023] Input the first image and the first sampled image into the backbone network of the teacher model and extract features, and use the region prediction head RoIHead to generate classification probability and bounding box regression parameters to obtain the second loss function;

[0024] Based on the first loss function and the second loss function, a multi-scale regularization loss is obtained.

[0025] Furthermore, based on the teacher model, the unlabeled data is screened by the re-prediction method and the bounding box regression loss is calculated to obtain the method of unsupervised positioning loss, including:

[0026] Input the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead of the teacher model again to obtain a second prediction result; wherein the second prediction result includes the frame coordinates and category score of the second target candidate box;

[0027] In response to the category score in the second prediction result being greater than a second threshold, acquiring a first label;

[0028] Based on the first label and the first prediction result, obtaining an intersection-over-union ratio of a border in the first label and a border in the first prediction result;

[0029] In response to the intersection-over-union ratio being greater than a third threshold, obtaining a reliable positioning pseudo label;

[0030] Obtain unsupervised localization loss based on reliable localization pseudo-labels.

[0031] The beneficial effects of the present application are as follows: the present application trains the model through a small amount of labeled data and a large amount of unlabeled data, and finally achieves good detection accuracy, which can effectively reduce the labor cost of data annotation. In addition, the present application proposes a multi-scale regularization strategy and a re-prediction algorithm, the former effectively improves the robustness of the model, and the latter further improves the positioning accuracy of the model. Finally, the present application performs data enhancement and adjusts the loss function, which mainly affects the amount of calculation in the model training stage, and has no effect on the amount of calculation of the model in the inference stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of an embodiment of a semi-supervised target detection method of the present application;

[0033] Figure 2 for Figure 1 A schematic flow chart of an embodiment of step S3 in FIG.

[0034] Figure 3 A schematic diagram of using multi-scale regularization on RPN in an embodiment of a semi-supervised target detection method of the present application;

[0035] Figure 4 for Figure 2 A schematic flow chart of an embodiment of step S32 in FIG. 1 ;

[0036] Figure 5 A schematic diagram of using multi-scale regularization on RoIHead in an embodiment of a semi-supervised target detection method;

[0037] Figure 6 for Figure 1 A schematic flow chart of an embodiment of step S4 in FIG.

[0038] Figure 7 This is the detection effect diagram of the original model;

[0039] Figure 8 This is a detection model rendering of the semi-supervised target detection method of this application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.

[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0042] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a semi-supervised target detection method of the present application. The method comprises the following steps:

[0043] Step S1. Divide the image data set into labeled data and unlabeled data, and perform data enhancement on the labeled data and the unlabeled data; wherein the amount of unlabeled data is greater than that of labeled data.

[0044] Specifically, the image dataset is divided into labeled data and unlabeled data. Usually, the amount of unlabeled data is much larger than that of labeled data.

[0045] Multi-scale scaling is used for data enhancement on labeled and unlabeled data, randomly scaling the image width and height in equal proportions, with the scaling range of -50% to 150%. During the training process, the input images with constantly changing sizes enable the model to learn the features of targets of various sizes, making the model more robust when detecting targets of different sizes.

[0046] Step S2. Based on the two-stage target detection algorithm Faster R-CNN, obtain the teacher model and the student model.

[0047] Specifically, a teacher-student model framework is used. Both the teacher and student models adopt the two-stage target detection algorithm Faster R-CNN. The teacher model is responsible for predicting unlabeled data and screening out reliable classification pseudo labels for calculating classification loss through the threshold method. The student model learns and updates parameters based on the label information of labeled data and the pseudo label information of the teacher model.

[0048] Step S3. Based on the teacher model, the unlabeled data is screened by the threshold method and the classification loss is calculated to obtain the multi-scale regularization loss and reliable classification pseudo labels.

[0049] For details, see Figure 2-3 , Figure 2 for Figure 1 A schematic flow chart of an embodiment of step S3 in FIG. Figure 3 A schematic diagram of using multi-scale regularization on RPN (Region Proposal Network) in one embodiment of the semi-supervised target detection method of the present application.

[0050] Step S31. Based on inputting the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead (Region of Interest Head) of the teacher model, a first prediction result is obtained; wherein the first prediction result includes the border coordinates and category score of the target candidate box.

[0051] Specifically, the first image in the unlabeled data is downsampled by maximum pooling, and the constrained image and the corresponding downsampled image are constrained so that their output results on the region candidate network RPN and the region prediction head RoIHead are consistent to obtain the first prediction result. All sampling methods in this embodiment are maximum pooling methods.

[0052] Step S32: constrain the output results of the region candidate network RPN and the region prediction head RoIHead to obtain a multi-scale regularization loss.

[0053] For details, see Figure 4-5 , Figure 4 for Figure 2 A schematic flow chart of an embodiment of step S32 in FIG. 1 ; Figure 5 This is a schematic diagram of using multi-scale regularization on RoIHead in an embodiment of a semi-supervised object detection method. Step S32 includes:

[0054] Specifically, the first image img in the unlabeled data and the first sampled image img after maximum pooling downsampling are d Input the backbone network to extract features, and use the regional proposal network RPN to generate img candidate boxes and category scores as well as img d Candidate boxes and category scores.

[0055] Step S321: Perform maximum pooling down-sampling on the first image in the unlabeled data to obtain a first sampled image.

[0056] Specifically, the first image img is subjected to maximum pooling downsampling to obtain a first sampled image imgd .

[0057] Step S322. Input the first image and the first sampled image into the backbone network of the teacher model and extract features, and use the region candidate network RPN to generate a first candidate box and obtain a first loss function.

[0058] Specifically, the first image img and the first sampled image img d Input into the backbone network of the teacher model and extract features, and use the regional proposal network RPN to generate candidate boxes.

[0059] The probability map obj output by the first image img on the region candidate network RPN is sampled by maximum pooling to obtain the first sampled map img d Probability map obj with the same size as the probability map * , and calculate the img probability map obj according to the following formula (1) * with img d Probability map obj d The distance between Wherein formula (1) is:

[0060]

[0061] in, represents the first loss function, obj[i] represents the i-th probability map output by the image on the region proposal network RPN, and m is the number of probability maps (for an image, the backbone network outputs five feature maps of different scales, each feature map will generate a probability map, so m = 5).

[0062] Step S323. Input the first image and the first sampled image into the backbone network of the teacher model and extract features, and use the region prediction head RoIHead to generate classification probability and bounding box regression parameters to obtain the second loss function.

[0063] Specifically, the first image img and the first sampled image img after maximum pooling downsampling d Input the backbone network to extract features, and the region candidate network RPN generates candidate boxes based on the feature map of img, where img d Use the same candidate box as img.

[0064] Img d The candidate region features of and img are respectively input into the region prediction head RoIHead to obtain the classification probability and bounding box regression parameters of the candidate region, and the second loss function is calculated according to the following formula (2). Formula (2) is

[0065]

[0066] Among them, cls and reg are the candidate box region classification probability and candidate box border regression parameters of the first image img, cls d and reg d For the first sample image img d The candidate box region classification probability and candidate box border regression parameters, cls i and reg i They represent the classification probability and bounding box regression parameter of the i-th candidate box in an image respectively. N is the total number of candidate boxes in an image, where N=1000.

[0067] Step S324. Based on the first loss function and the second loss function, obtain a multi-scale regularization loss.

[0068] Specifically, as shown in formula (3), the loss of unlabeled data on the region candidate network RPN and the region prediction head RoIHead is added as the multi-scale regularization loss L scale . Formula (3) is:

[0069]

[0070] Step S33: In response to the highest category score of the target candidate box being greater than a first threshold, obtaining a reliable classification pseudo label.

[0071] Specifically, according to the first prediction result, a threshold method is used to take the target frame in the first image whose highest category score is greater than the first threshold as a reliable classification pseudo label, wherein the first threshold is a category determination threshold.

[0072] Step S4. Based on the teacher model, the unlabeled data is screened by the re-prediction method and the bounding box regression loss is calculated to obtain the unsupervised positioning loss.

[0073] Specifically, the above pseudo labels only ensure that the classification of the target is reliable, but cannot ensure that the corresponding bounding box coordinates are reliable. Therefore, these pseudo labels can only calculate the classification loss on unlabeled data. Unable to calculate bounding box regression loss The accuracy of the positioning result of the teacher model is determined by the re-prediction algorithm, and the reliable classification pseudo-labels with accurate positioning are used to calculate the positioning loss of the unlabeled data of the student model.

[0074] See also Figure 6 , Figure 6 for Figure 1 Schematic diagram of the process of step S4 in an embodiment. Step S4 includes:

[0075] Step S41. Input the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead of the teacher model again to obtain a second prediction result; wherein the second prediction result includes the frame coordinates and category score of the second target candidate box. The first image in the unlabeled data includes: unlabeled data and pseudo labels corresponding to the unlabeled data (pseudo labels that have not been screened by any threshold).

[0076] Specifically, the first image in the unlabeled data is input into the teacher model, and the region candidate network RPN of the teacher model predicts the candidate region. Then the region prediction head RoIHead of the teacher model performs a classification and regression operation on the candidate region to obtain the first prediction result (including bounding box coordinates and category scores).

[0077] Based on the bounding box B0 of the first prediction result, the regional prediction head RoIHead of the teacher model is used for re-prediction to obtain the bounding box B of the second prediction result. tmp (Contains bounding box coordinates and category scores).

[0078] Step S42: In response to the category score in the second prediction result being greater than the second threshold, obtaining the first label.

[0079] Specifically, the border B of the second prediction result tmp The foreground objects whose highest category scores are greater than the second threshold τ1 (τ1=0.3 in this embodiment) are screened out and put into the first label. The second threshold is the foreground-background determination threshold.

[0080] Step S43: Based on the first label and the first prediction result, obtain an intersection-over-union ratio of the bounding box in the first label and the bounding box in the first prediction result.

[0081] Specifically, get B tmp The Intersection over Union (IoU) with each bounding box in B0.

[0082] Step S44: In response to the intersection-over-union ratio being greater than a third threshold, obtaining a reliable positioning pseudo label.

[0083] Specifically, B0 and B tmp The frames in are compared one by one, and if the intersection-and-union ratio is greater than the third threshold τ2 (τ2=0.95 in this embodiment), it is considered that the frame coordinates are accurately positioned, and the frame coordinates are stored in the reliable positioning pseudo label B'. The third threshold is the positioning threshold.

[0084] Step S45: Obtain unsupervised positioning loss based on the reliable positioning pseudo-label.

[0085] Specifically, based on the above-mentioned reliable classification pseudo labels and reliable positioning pseudo labels, the classification loss of unlabeled data is calculated respectively. and the localization loss of unlabeled data The classification loss is calculated by the cross entropy loss function, and the regression loss is calculated by the SmoothL1 loss function. The final unsupervised localization loss L is obtained according to formula (4): unsup .

[0086] Formula (4) is:

[0087]

[0088] Step S5. Obtain supervised loss based on the student model and labeled data.

[0089] Specifically, the supervised loss calculated based on the labeled data using the label information includes classification loss and regression loss. The classification loss is calculated using the cross entropy loss function, and the regression loss is calculated using the SmoothL1 loss function. The supervised loss L is calculated according to formula (5): sup .

[0090] Formula (5) is:

[0091]

[0092] Step S6. Obtain the final model loss function based on the multi-scale regularization loss, the unsupervised positioning loss, and the supervised loss.

[0093] Specifically, according to formula (6), the final model loss function is calculated. Formula (6) is:

[0094] L=L sup +αL unsup +βL scale (6);

[0095] Among them, L sup It is a supervised loss calculated based on labeled data using label information, including classification loss and regression loss. The classification loss is calculated using the cross entropy loss function, and the regression loss is calculated using the Smooth L1 loss function. unsup It is the unsupervised positioning loss calculated based on unlabeled data using pseudo-label information. The specific calculation is the same as L sup Consistent. scale It is the multi-scale regularization loss proposed by the present invention, α and β are weights, and in this embodiment, α=2, β=1.

[0096] Step S7. The student model updates the first parameter by gradient descent based on the final model loss function; the teacher model updates the second parameter by EMA method based on the first parameter.

[0097] Specifically, the student model updates parameters using gradient descent according to the loss function, and the teacher model updates parameters using the EMA method based on the student model parameters.

[0098] Table 1 is a performance comparison table of this application and other semi-supervised object detection algorithms on the COCO dataset.

[0099]

[0100] As shown in Table 1, in the COCO dataset, randomly selected datasets are selected, of which the labeled data accounts for 1%, 2%, 5%, and 10%, respectively, and the rest is unlabeled data. Under the setting of 1%, 2%, 5%, and 10% labeled data, the mAP (meanAverage Precision) of the present invention still reaches 21.21%, 25.49%, 29.02%, and 31.84% respectively. Therefore, the present application can effectively reduce the labor cost of data annotation.

[0101] like Figure 7-8 As shown, Figure 7 This is the detection effect diagram of the original model; Figure 8 This is a detection model rendering of the semi-supervised target detection method of this application.

[0102] Depend on Figure 7-8 The detection results can be visualized (the original baseline model has problems of false detection, repeated detection and missed detection (marked with "white" arrows). The improved algorithm of this application effectively solves these problems. At the same time, the positioning result of the target is more accurate (such as Figure 8 b, 8c).

[0103] This application is a semi-supervised target detection algorithm that uses a small amount of labeled data and a large amount of unlabeled data to train the model, and ultimately achieves good detection accuracy. In addition, this application proposes a multi-scale regularization strategy and a re-prediction algorithm, the former effectively improves the robustness of the model, and the latter further improves the positioning accuracy of the model. Finally, the present invention performs data enhancement and adjusts the loss function, which mainly affects the amount of computation in the model training phase and has no effect on the amount of computation in the model inference phase.

[0104] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A semi-supervised target detection method, characterized in that: The following steps are involved: The image data set is divided into labeled data and unlabeled data, and data enhancement is performed on the labeled data and the unlabeled data; wherein the amount of the unlabeled data is greater than that of the labeled data; Based on the two-stage target detection algorithm Faster R-CNN, obtain the teacher model and student model; Based on the teacher model, the unlabeled data is screened by a threshold method and the classification loss is calculated to obtain a multi-scale regularization loss and a reliable classification pseudo label; specifically, the method includes: Based on inputting the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead of the teacher model, obtaining a first prediction result; wherein the first prediction result includes the frame coordinates and category score of the target candidate box; Constraining the output results of the region candidate network RPN and the region prediction head RoIHead to obtain a multi-scale regularization loss; In response to the highest category score of the target candidate box being greater than a first threshold, acquiring the reliable classification pseudo label; Perform maximum pooling downsampling on the first image in the unlabeled data to obtain a first sampled image; The first image and the first sampled image are input into the backbone network of the teacher model to extract features, and a region candidate network RPN is used to generate a first candidate box and obtain a first loss function; wherein, based on formula (1), the first loss function is obtained; the formula (1) is: （1）； in, represents the first loss function, Represents the output of the image on the region proposal network RPN i A probability map, m is the number of probability maps; is a probability map of the first image; is a probability map of the first sampled image; The first image and the first sampled image are input into the backbone network of the teacher model to extract features, and the regional prediction head RoIHead is used to generate classification probability and bounding box regression parameters to obtain a second loss function; wherein, based on formula (2), the second loss function is obtained; the formula (2) is: （2）； in, is the second loss function, cls and reg are the classification probability of the candidate box area and the regression parameters of the candidate box border of the first image img, cls d and reg d For the first sample image img d The candidate box region classification probability and candidate box border regression parameters, cls i and reg i Respectively represent an image i The classification probability and bounding box regression parameters of candidate boxes, N is the total number of candidate boxes in an image; Based on the first loss function and the second loss function, the multi-scale regularization loss is obtained; wherein, based on formula (3), the multi-scale regularization loss is obtained; the formula (3) is: （3）； Based on the teacher model, the unlabeled data is screened by a re-prediction method and the bounding box regression loss is calculated to obtain an unsupervised positioning loss; wherein the method for obtaining the unsupervised positioning loss includes: Inputting the first image in the unlabeled data into the region candidate network RPN and the region prediction head RoIHead of the teacher model again to obtain a second prediction result; wherein the second prediction result includes the frame coordinates and category score of the second target candidate box; In response to the category score in the second prediction result being greater than a second threshold, acquiring a first label; Based on the first label and the first prediction result, obtaining an intersection-over-union ratio of a border in the first label and a border in the first prediction result; In response to the intersection-over-union ratio being greater than a third threshold, obtaining a reliable positioning pseudo label; Based on the reliable positioning pseudo-label and the reliable classification pseudo-label, an unsupervised positioning loss is obtained; wherein, based on formula (4), the unsupervised positioning loss is obtained; the formula (4) is: （4）； in, is the unsupervised localization loss, is the classification loss for unlabeled data and is the localization loss of unlabeled data; Obtaining a supervised loss based on the student model and the labeled data; Based on the multi-scale regularization loss, the unsupervised positioning loss and the supervised loss, obtaining a final model loss function; The student model updates the first parameter by gradient descent based on the final model loss function; and the teacher model updates the second parameter by EMA method based on the first parameter.

2. The method according to claim 1, characterized in that The method for performing data enhancement on the labeled data and the unlabeled data comprises: Multi-scale scaling is used to perform data enhancement on the unlabeled data and the labeled data, and the width and height of the image are randomly scaled in equal proportions.

3. The method according to claim 2, characterized in that The range of the proportional scaling is -50% to 150%.

Images