Evaluation method, screening method, training method and device for annotation model

Abstract

The present disclosure provides an evaluation method, a screening method, a training method and a device for an annotation model. The evaluation method includes using the annotation model to be evaluated to predict the bounding box of the detection target of each sample image in a preset validation set, so as to obtain the predicted bounding box of each sample image; calculating the margin difference of each side between the predicted bounding box of each sample image and the preset true bounding box; and evaluating the annotation model to be evaluated based on each margin difference. The present disclosure evaluates the annotation model to be evaluated from the perspective of the annotator and from the perspective of the boundary differences between the predicted bounding box and the true bounding box, so that the evaluation result conforms to the business requirements of the manual annotation scenario.

Description

Technical Field

[0001] The present disclosure relates to the technical field of object detection, and in particular, to an evaluation method, a screening method, a training method, and a device for an annotation model. Background Art

[0002] As a common task in computer vision, object detection has been widely explored and applied. Before performing object detection, it is necessary to first annotate the detection targets in the training images to construct the training data of the object detection model, and then train an intelligent model that can perform object detection through the training data.

[0003] In the scenario of manual annotation, when annotating the detection targets in the training images, usually the initial types and initial bounding boxes of the detection targets in the training images are first annotated by the selected annotation model. Then, the annotator judges whether the initial types and initial bounding boxes are correct, and revises the incorrect annotation types and bounding boxes according to the judgment results. Therefore, the selected annotation model will directly affect the workload and work efficiency of the annotator.

[0004] Currently, when screening an annotation model, the annotation model can be accurately evaluated from aspects such as classification, overlapping area, and performance. However, the current evaluation methods cannot meet the business requirements of the manual annotation scenario. For example, when evaluating the annotation model through the overlapping area, the overlapping degree between the predicted bounding box output by the annotation model and the true bounding box will be accurately evaluated. However, in the annotation scenario, even if the overlapping degree is high, but the margin difference is greater than the tolerance of the annotator, it is still regarded as an equal failure, and the annotator needs to make a manual correction. Therefore, the method for evaluating the annotation model based on the overlapping area cannot meet the business requirements of the manual annotation scenario. Summary of the Invention

[0005] In view of this, the present disclosure provides an evaluation method, a screening method, a training method, and a device for an annotation model, which evaluate the annotation model from the perspective of the annotator, so that the evaluation result is more in line with the business requirements of manual annotation.

[0006] According to a first aspect of the present disclosure, there is provided an evaluation method for an annotation model, including:

[0007] Using the annotation model to be evaluated, predict the bounding boxes of the detection targets in each sample image in the preset validation set to obtain the predicted bounding boxes of each sample image;

[0008] Calculate the margin differences of each side between the predicted bounding box of each sample image and the preset true bounding box;

[0009] Based on the margin differences, evaluate the annotation model to be evaluated.

[0010] In a possible implementation, when calculating the margin differences between the predicted bounding box and the ground truth bounding box, it includes:

[0011] Obtain the position information of the predicted bounding box and the position information of the ground truth bounding box;

[0012] Calculate the margin differences of each boundary according to the position information of the predicted bounding box and the position information of the ground truth bounding box.

[0013] In a possible implementation, when evaluating the annotation model to be evaluated based on the margin differences, it includes:

[0014] Calculate the number of bounding box adjustments on the validation set based on the margin differences;

[0015] Determine the evaluation result of the annotation model to be evaluated according to the number of bounding box adjustments on the validation set.

[0016] In a possible implementation, when calculating the number of bounding box adjustments on the validation set based on the margin differences, it includes:

[0017] Determine the magnitude relationship between each margin difference and a preset margin difference threshold;

[0018] Calculate the adjustment times corresponding to each margin difference according to the magnitude relationship between each margin difference and the preset margin difference threshold;

[0019] Obtain the number of bounding box adjustments on the validation set according to the adjustment times corresponding to each margin difference.

[0020] According to the second aspect of the present disclosure, a method for screening an annotation model is provided, including:

[0021] Obtain at least two annotation models;

[0022] Evaluate the at least two annotation models by using the evaluation method described in any one of the first aspect to obtain the evaluation results of the at least two annotation models;

[0023] Select a target annotation model from the at least two annotation models based on the evaluation results.

[0024] According to the third aspect of the present disclosure, a method for training an annotation model is provided, including:

[0025] Use the annotation model to be trained to predict the bounding box of the detection target in the preset sample image to obtain the predicted bounding box of the sample image;

[0026] Calculate the margin differences between the predicted bounding box of the sample image and the preset ground truth bounding box;

[0027] Calculate a loss function revision function based on the respective margin differences;

[0028] Train the to-be-trained annotation model based on the loss function revision function.

[0029] In a possible implementation manner, the calculation formula of the loss function revision function is as follows:

[0030]

[0031] In the formula, m is the revision value, k is the margin difference tolerance, β is the scaling factor, and x is the margin difference.

[0032] According to a fourth aspect of the present disclosure, there is provided an evaluation device for an annotation model, including:

[0033] A first prediction module, configured to use the to-be-evaluated annotation model to perform bounding box prediction on the detection targets of each sample picture in a preset validation set, and obtain the predicted bounding boxes of each of the sample pictures;

[0034] A first margin difference calculation module, configured to calculate the respective margin differences between the predicted bounding boxes of each of the sample pictures and the preset true bounding boxes;

[0035] An evaluation module, configured to evaluate the to-be-evaluated annotation model based on the respective margin differences.

[0036] According to a fifth aspect of the present disclosure, there is provided a screening device for an annotation model, including:

[0037] A model acquisition module, configured to acquire at least two annotation models;

[0038] A model evaluation module, configured to use any one of the evaluation methods in the first aspect to evaluate the at least two annotation models, and obtain the evaluation results of the at least two annotation models;

[0039] A model screening module, configured to screen out a target annotation model from the at least two annotation models based on the evaluation results.

[0040] According to a sixth aspect of the present disclosure, there is provided a training device for an annotation model, including:

[0041] A second prediction module, configured to use the to-be-trained annotation model to perform bounding box prediction on the detection target of a preset sample picture, and obtain the predicted bounding box of the sample picture;

[0042] A second margin difference calculation module, configured to calculate the respective margin differences between the predicted bounding box of the sample picture and the preset true bounding box;

[0043] A revised value construction module for calculating a loss function revision function based on the margin differences.

[0044] A training module for training the annotation model to be trained based on the loss function revision function.

[0045] In the present disclosure, from the perspective of an annotator, the annotation model to be evaluated is evaluated from the perspective of the margin differences between the predicted bounding box and the true bounding box, so that the evaluation result conforms to the business requirements of the manual annotation scenario.

[0046] Other features and aspects of the present disclosure will become apparent from the following detailed description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings included in and constituting a part of this specification, together with the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0048] Figure 1 A schematic flowchart showing an annotation model evaluation method according to an embodiment of the present disclosure;

[0049] Figure 2 A schematic flowchart showing an annotation model screening method according to an embodiment of the present disclosure;

[0050] Figure 3 A schematic flowchart showing an annotation model training method according to an embodiment of the present disclosure;

[0051] Figure 4 A schematic diagram showing a loss function revision function according to an embodiment of the present disclosure;

[0052] Figure 5 A schematic diagram showing a loss function revision function according to another embodiment of the present disclosure;

[0053] Figure 6 A schematic block diagram showing an annotation model evaluation device according to an embodiment of the present disclosure;

[0054] Figure 7 A schematic block diagram showing an annotation model screening device according to an embodiment of the present disclosure;

[0055] Figure 8 A schematic block diagram showing an annotation model training device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote functionally identical or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0057] As used herein, the term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0058] In addition, for a better description of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0059] <Embodiment of the evaluation method>

[0060] Figure 1 A schematic flowchart showing a method for evaluating an annotation model according to an embodiment of the present disclosure is shown. As Figure 1 shown, the method for evaluating the annotation model includes steps S1100 - S1300.

[0061] S1100, using the annotation model to be evaluated, predict the bounding boxes of the detection targets in each sample image in the preset validation set, and obtain the predicted bounding boxes of each sample image.

[0062] The annotation model to be evaluated is the annotation model to be evaluated currently. The annotation model can predict the type and bounding box of the detection target in the image, and mark the prediction result on the image, that is, the prediction result includes the predicted type and predicted bounding box of the detection target. The annotation model can be a Yolo model, an R - CNN model, or other models that can realize the prediction of the detection target in the image, and no specific limitation is made here.

[0063] The validation set is used to test the prediction performance of the annotation model to be evaluated. The validation set includes multiple images including detection targets. Among them, for each image in the validation set, there is a corresponding true annotation result preset. The true annotation result includes the true type and true bounding box of the detection target. Each image can be corresponding to its corresponding true annotation result through the IP address of the image. In this way, the true annotation result corresponding to the image can be obtained through the IP address of the image.

[0064] When using the annotation model to be evaluated to predict the bounding boxes of the detection targets in each sample image in the preset validation set, the annotation model to be evaluated will predict the detection target types and bounding boxes for each image in the validation set, obtain the prediction results corresponding to each image, and extract the predicted bounding boxes of the detection targets from the prediction results. Among them, each image and its corresponding prediction result can be corresponded through the IP address of the image. In this way, the prediction result corresponding to the image can be obtained through the IP address of the image, and thus the predicted bounding boxes of the detection targets can be extracted from the prediction results.

[0065] S1200, calculate the margin differences of each side between the predicted bounding boxes of each sample image and the preset true bounding boxes. Taking one image as an example, step S1200 will be described below.

[0066] After obtaining the predicted bounding boxes of the detection targets in the image, the true annotation result corresponding to the image can be obtained according to the IP address of the image, and the true bounding boxes can be extracted from the true annotation result.

[0067] When calculating the margin differences of each side between the predicted bounding boxes and the true bounding boxes, steps S1210 - S1220 can be included.

[0068] S1210, obtain the position information of the predicted bounding box and the position information of the true bounding box.

[0069] In a possible implementation, the predicted bounding box and the true bounding box can be square or rectangular, and no specific limitation is made here.

[0070] In this implementable manner, the position information of the predicted bounding box can include the pixel coordinates (x1, y1) of the upper left corner and the pixel coordinates (x2, y2) of the lower right corner of the predicted bounding box. The position information of the true bounding box can include the pixel coordinates (x3, y3) of the upper left corner and the pixel coordinates (x4, y4) of the lower right corner of the true bounding box. In this way, the margin differences of each side between the predicted bounding box and the true bounding box can be calculated through the pixel coordinates of the predicted bounding box and the true bounding box.

[0071] S1220, calculate the margin differences of each side according to the position information of the predicted bounding box and the position information of the true bounding box.

[0072] In the above embodiment, the margin difference of the upper border between the predicted bounding box and the true bounding box can be calculated by |y1 - y3|. The margin difference of the lower border between the predicted bounding box and the true bounding box can be calculated by |y2 - y4|. The margin difference of the left border between the predicted bounding box and the true bounding box can be calculated by |x1 - x3|. The margin difference of the right border between the predicted bounding box and the true bounding box can be calculated by |x2 - x4|.

[0073] Referring to steps S1210 - S1220, the margin differences between the predicted bounding boxes and the preset true bounding boxes of each sample image can be calculated, which will not be elaborated here.

[0074] S1300. Evaluate the annotation model to be evaluated based on the margin differences.

[0075] In a possible implementation, when evaluating the annotation model to be evaluated based on the margin differences, steps S1310 - S1320 are included.

[0076] S1310. Calculate the number of bounding box adjustments on the validation set based on the margin differences.

[0077] It should be noted that in the manual annotation scenario, the annotator will set a margin difference threshold according to the user's tolerance for the margin difference. When the margin difference between the predicted bounding box and any side border of the true bounding width is greater than the margin difference threshold, it means that the annotation result of this side's predicted border has exceeded the user's acceptance range. Therefore, the annotator needs to make a manual adjustment to this side's predicted border so that the annotation result of this side's predicted border meets the user's tolerance.

[0078] Based on the above concept, in a possible implementation, when calculating the number of bounding box adjustments on the validation set based on the margin differences, steps S1311 - S1313 are included.

[0079] S1311. Determine the magnitude relationship between each margin difference and the preset margin difference threshold.

[0080] Specifically, the margin difference threshold can be set according to the user's specific needs. For example, the margin difference threshold can be set to 4 pixel values, i.e., 4px, according to the user's specific needs.

[0081] S1312. Calculate the adjustment times corresponding to each margin difference according to the magnitude relationship between each margin difference and the preset margin difference threshold. Specifically, the calculation formula for the adjustment times corresponding to the margin difference can be as shown in formula (1):

[0082]

[0083] In the formula, n is the adjustment times, d is the margin difference, and k is the margin difference threshold.

[0084] When calculating the number of adjustment times corresponding to the margin difference by formula (1), if k is 4px and the margin difference between the upper border of the predicted bounding box and the ground truth bounding box is 3px, at this time 3px is less than 4px. According to formula (1), the number of adjustment times corresponding to the margin difference of the upper border is 0, that is, no adjustment by the annotator is required. The margin difference between the left border of the predicted bounding box and the ground truth bounding box is 5px, at this time 5px is greater than 4px. According to formula (1), the number of adjustment times corresponding to the margin difference of the left border is 1, that is, the annotator needs to make 1 adjustment to the predicted border on the left.

[0085] Referring to step S1312, the number of adjustment times corresponding to each margin difference can be calculated, which will not be elaborated here.

[0086] S1313. Obtain the number of bounding box adjustment times on the validation set according to the number of adjustment times corresponding to each margin difference.

[0087] In the implementable manner where the predicted bounding box and the ground truth bounding box can be square or rectangular, for the predicted bounding box of each image, there are 4 borders: upper, lower, left, and right. Therefore, it is necessary to first sum up the number of adjustment times corresponding to the 4 borders to obtain the number of bounding box adjustment times for each image. Then, sum up the number of bounding box adjustment times for each image in the validation set to obtain the number of bounding box adjustment times on this validation set.

[0088] S1320. Determine the evaluation result of the annotation model to be evaluated according to the number of bounding box adjustment times on the validation set. Specifically, the number of bounding box adjustment times on the validation set reflects the prediction effect of the annotation model to be evaluated in the manual annotation scenario. Therefore, in one possible implementation manner, the number of bounding box adjustment times on the validation set can be directly used as the evaluation value of the annotation model to be evaluated, so as to determine the evaluation result of the model to be evaluated through this evaluation value.

[0089] It should be noted that the larger the evaluation value, the more times the annotator needs to manually adjust, indicating that the prediction result of the model to be evaluated for the bounding box does not meet the needs of the annotator. On the contrary, the smaller the evaluation value, the fewer times the annotator needs to manually adjust, indicating that the prediction result of the model to be evaluated for the bounding box is more in line with the needs of the annotator. Therefore, evaluating the model to be evaluated through the method of the present disclosure can make the evaluation result meet the business requirements of the manual annotation scenario.

[0090] <Embodiment of the screening method>

[0091] Figure 2 Show a schematic flowchart of an annotation model screening method according to an embodiment of the present disclosure. As Figure 2 shown, the annotation model screening method includes steps S2100 - S2300.

[0092] S2100, obtain at least two annotation models.

[0093] It should be noted that before obtaining the annotation models, it is necessary to first select the base models of the annotation models, determine the configurable hyperparameters of the base models according to the algorithms of the base models. Then, according to the configurable hyperparameters, determine at least two hyperparameter combinations. After determining at least two hyperparameter combinations, configure the corresponding base models respectively according to each hyperparameter combination. Finally, use the preset training samples to train the base models corresponding to each hyperparameter combination to obtain at least two annotation models.

[0094] For example, the algorithm of the base model is the grid search method. Suppose the base model contains two hyperparameters A and B, A examines three values (a1, a2, a3), and B examines two values (b1, b2). After exhaustive enumeration, 6 hyperparameter combinations (a1b1, a1b2, a2b1, a2b2, a3b1, a3b2) are generated. Configure the base model A1 according to the hyperparameter combination a1b1, configure the base model A2 according to the hyperparameter combination a1b2, configure the base model A3 according to the hyperparameter combination a2b1, configure the base model A4 according to the hyperparameter combination a2b2, configure the base model A5 according to the hyperparameter combination a3b1, and configure the base model A6 according to the hyperparameter combination a3b2. Finally, use the preset training samples to train the base models A1 - A6 respectively to obtain a total of 6 annotation models.

[0095] S2200, use any evaluation method in the evaluation method embodiments to evaluate at least two annotation models to obtain the evaluation results of at least two annotation models.

[0096] Continuing with the above embodiment, use any one of the evaluation methods in the evaluation method embodiments to calculate the evaluation values of the 6 annotation models respectively, and use the evaluation values of each annotation model as the evaluation results of each annotation model.

[0097] S2300, based on the evaluation results, select the target annotation model from at least two annotation models.

[0098] Continuing with the above embodiment, the annotation model with the lowest evaluation value can be selected from the 6 annotation models as the target annotation model.

[0099] In this embodiment, screening the annotation models based on the evaluation methods in the evaluation method embodiments can make the selected target annotation model more in line with the business requirements in the manual annotation scenario.

[0100] Furthermore, annotating the detection targets in the pictures based on the target annotation model in this embodiment can reduce the number of times of adjusting the prediction bounding boxes by the annotators, reduce the workload of the annotators, and improve the annotation efficiency.

[0101] In this embodiment, the hyperparameter combination corresponding to the target annotation model can also be recorded as the hyperparameter combination that meets the requirements of the manual annotation scenario, so as to accurately select the hyperparameter combination of the annotation model through the method of this embodiment.

[0102] <Embodiment of the training method>

[0103] Figure 3 The schematic flowchart showing the annotation model training method according to an embodiment of the present disclosure is shown.

[0104] It should be noted that the training process of each picture in the training set for the basic model is the same. Therefore, taking one picture in the training set as an example, the training method of this embodiment will be described exemplarily.

[0105] As Figure 3 shown, the annotation model training method includes steps S3100-S3400.

[0106] S3100, using the annotation model to be trained, predict the bounding box of the detection target of the preset sample picture, and obtain the predicted bounding box of the sample picture.

[0107] The annotation model to be trained is the basic model of the annotation model to be trained currently. By training the annotation model to be trained with the sample pictures in the preset training set, an annotation model that can predict the detection target in the picture can be obtained.

[0108] The preset sample picture is the sample picture in the preset training set. For each picture in the training set (different from the pictures in the validation set), there is a corresponding true annotation result preset. The true annotation result is similar to the true annotation result corresponding to each picture in the validation set, which will not be elaborated here.

[0109] When using the annotation model to be trained to predict the bounding box of the detection target of the sample picture, the annotation model to be trained will detect the type of the detection target and predict the bounding box for the detection target in the sample picture, obtain the prediction result corresponding to the sample picture, and extract the predicted bounding box of the detection target from the prediction result.

[0110] S3200, calculate the margin difference of each side between the predicted bounding box of the sample picture and the preset true bounding box.

[0111] Specifically, for the steps of calculating the margin difference, refer to step S1200, which will not be elaborated here.

[0112] S3300, based on each margin difference, calculate the loss function revision function.

[0113] In a possible implementation, the calculation formula of the loss function revision function is shown in formula (2):

[0114]

[0115] In the formula, m is the revision value, k is the margin difference tolerance, β is the scaling factor, and x is the margin difference. Among them, the margin difference tolerance is a multiple of the margin difference threshold, and this multiple can be set according to specific requirements. For example, this multiple can be 2, that is, the margin difference tolerance k is 2 times the margin difference threshold. The scaling factor β is used to control the sensitivity of the function to the margin, and the scaling factor β can be set according to the confidence in the margin difference tolerance k. The higher the confidence, the larger the value of β. For example, when the confidence is 90%, the scaling factor can be 90. When the confidence is 10%, the scaling factor can be 10. Among them, the scaling factor is preferably set within the range of less than or equal to 100.

[0116] For example, when k = 6 (used to simulate the margin difference of 3 pixels) and β = 90, the calculated loss function revision function can be as Figure 4 shown.

[0117] Another example, when k = 6 (used to simulate the margin difference of 3 pixels) and β = 10, the calculated loss function revision function can be as Figure 5 shown.

[0118] In this implementable manner, when obtaining the margin difference between the predicted bounding box and one side of the true bounding box, substituting this margin difference into formula (2), the sub-loss function revision function corresponding to the side border can be obtained. When using this method to obtain the sub-loss function revision functions corresponding to each border of the predicted bounding box, summing up the sub-loss function revision functions corresponding to each border can complete the construction of the final loss function revision function.

[0119] S3400. Train the annotation model to be trained based on the loss function revision function.

[0120] Specifically, the loss function revision function and the loss function of the model to be trained can be summed up first to complete the adjustment of the loss function, and then the training of the model to be trained is guided by the adjusted loss function.

[0121] In the annotation model training method of this embodiment, first calculate the loss function revision function according to the requirements of the annotator for the margin difference, then adjust the loss function of the model to be trained through this loss function revision function, and finally guide the annotation model to train with the goal of reducing the margin difference through the adjusted loss function, so as to realize the effective training of the annotation model according to the requirements of the annotator for the margin difference, so that the trained annotation model better meets the business requirements of the manual annotation scenario.

[0122] Furthermore, when using the annotation model trained by the annotation model training method of this embodiment to annotate pictures, the number of adjustments made by the annotator to the predicted bounding box can be reduced, thereby reducing the workload of the annotator and improving the annotation efficiency of the detection target.

[0123] <Embodiment of the evaluation device>

[0124] Figure 6 The schematic block diagram of the annotation model evaluation device according to an embodiment of the present disclosure is shown. As Figure 6 shown, the annotation model evaluation device 100 includes:

[0125] A first prediction module 110, configured to use the annotation model to be evaluated to perform bounding box prediction on the detection target of each sample picture in the preset validation set, and obtain the predicted bounding box of each sample picture.

[0126] A first margin difference calculation module 120, configured to calculate the margin differences of each side between the predicted bounding box of each sample picture and the preset true bounding box.

[0127] An evaluation module 130, configured to evaluate the annotation model to be evaluated based on each margin difference.

[0128] <Embodiment of the screening device>

[0129] Figure 7 The schematic block diagram of the annotation model screening device according to an embodiment of the present disclosure is shown. As Figure 7 shown, the annotation model screening device 200 includes:

[0130] A model acquisition module 210, configured to acquire at least two annotation models.

[0131] A model evaluation module 220, configured to use the evaluation method of any one of the first aspects to evaluate at least two annotation models, and obtain the evaluation results of at least two annotation models.

[0132] A model screening module 230, configured to screen out the target annotation model from at least two annotation models based on the evaluation results.

[0133] <Embodiment of the training device>

[0134] Figure 8 The schematic block diagram of the annotation model training device according to an embodiment of the present disclosure is shown. As Figure 8 shown, the annotation model training device 300 includes:

[0135] A second prediction module 310, configured to use the annotation model to be trained to perform bounding box prediction on the detection target of the preset sample picture, and obtain the predicted bounding box of the sample picture.

[0136] The second margin difference calculation module 320 is configured to calculate the margin difference between the predicted bounding box of the sample image and the preset true bounding box.

[0137] The revision value construction module 330 is configured to calculate a loss function revision function based on the margin difference.

[0138] The training module 340 is configured to train the annotation model to be trained based on the loss function revision function.

[0139] The embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An evaluation method for an annotation model, characterized in that, Including: Using the annotation model to be evaluated, perform bounding box prediction on the detection targets of each sample image in the preset validation set to obtain the predicted bounding boxes of each said sample image; Calculate the margin differences of each side between the predicted bounding boxes of each said sample image and the preset true bounding boxes; Evaluate the annotation model to be evaluated based on the said margin differences; When evaluating the annotation model to be evaluated based on the said margin differences, including: Based on the said margin differences, calculate the number of bounding box adjustments on the validation set; Determine the evaluation result of the annotation model to be evaluated according to the number of bounding box adjustments on the validation set.

2. The method according to claim 1, characterized in that, When calculating the margin differences of each side between the predicted bounding box and the true bounding box, including: Obtain the position information of the predicted bounding box and the position information of the true bounding box; Calculate the margin differences of each boundary according to the position information of the predicted bounding box and the position information of the true bounding box.

3. The method according to claim 1, wherein When calculating the number of bounding box adjustments on the validation set based on the said margin differences, including: Determine the magnitude relationship between each margin difference and the preset margin difference threshold; Calculate the adjustment times corresponding to each margin difference according to the magnitude relationship between each margin difference and the preset margin difference threshold; Obtain the number of bounding box adjustments on the validation set according to the adjustment times corresponding to each margin difference.

4. A screening method for a labeling model, characterized in that, Including: Obtain at least two annotation models; Use the evaluation method described in any one of claims 1-3 to evaluate the at least two annotation models to obtain the evaluation results of the at least two annotation models; Based on the said evaluation results, screen out the target annotation model from the at least two annotation models.

5. A training method for an annotation model, characterized in that, Including: Using the annotation model to be trained, perform bounding box prediction on the detection target of the preset sample image to obtain the predicted bounding box of the sample image; Calculate the margin differences of each side between the predicted bounding box of the sample image and the preset true bounding box; Based on the said margin differences, calculate the loss function revision function; Train the annotation model to be trained based on the said loss function revision function; The loss function revision function is calculated based on the following formula: In the formula, m is the revised value, k is the margin difference tolerance, β is the scaling factor, x is the margin difference; When calculating the loss function revision function based on the said margin differences, calculate the margin differences of the corresponding sides between the predicted bounding box and the true bounding box, substitute the margin differences of the corresponding sides into the formula to obtain the sub-loss function revision functions of the corresponding sides, and sum up the sub-loss function revision functions of the corresponding sides to obtain the loss function revision function.

6. An evaluation device for an annotation model, characterized in that, Including: The first prediction module is used to use the annotation model to be evaluated to perform bounding box prediction on the detection targets of each sample image in the preset validation set to obtain the predicted bounding boxes of each said sample image; The first margin difference calculation module is used to calculate the margin differences of each side between the predicted bounding boxes of each said sample image and the preset true bounding boxes; The evaluation module is used to evaluate the annotation model to be evaluated based on the said margin differences; When the evaluation module evaluates the annotation model to be evaluated based on the said margin differences, it is specifically used for: Based on the said margin differences, calculate the number of bounding box adjustments on the validation set; Determine the evaluation result of the annotation model to be evaluated according to the number of times of adjusting the bounding box on the verification set.

7. A screening device for a labeling model, characterized in that, Including: A model acquisition module, configured to acquire at least two annotation models; A model evaluation module, configured to evaluate the at least two annotation models by using the evaluation method described in any one of claims 1-3, so as to obtain the evaluation results of the at least two annotation models; A model screening module, configured to screen out a target annotation model from the at least two annotation models based on the evaluation results.

8. A training device for a labeling model, characterized in that Including: A second prediction module, configured to use the annotation model to be trained to perform a bounding box prediction on the detection target of a preset sample picture, so as to obtain a predicted bounding box of the sample picture; A second margin difference calculation module, configured to calculate the margin differences of each side between the predicted bounding box of the sample picture and a preset true bounding box; A revised value construction module, configured to calculate a loss function revision function based on the margin differences of each side; A training module, configured to train the annotation model to be trained based on the loss function revision function; The loss function revision function is calculated based on the following formula: In the formula, m is the revised value, k is the margin difference tolerance, β is the scaling factor, x is the margin difference; When calculating the loss function revision function based on the margin differences of each side, calculate the corresponding margin differences of each side between the predicted bounding box and the true bounding box, substitute the corresponding margin differences of each side into the formula to obtain the sub-loss function revision functions corresponding to each side, and sum the sub-loss function revision functions corresponding to each side to obtain the loss function revision function.

Images