Method and device for determining labeling quality

Abstract

The present invention provides a method and device for determining labeling quality. The method comprises: obtaining an automatically labeled sample set based on all samples to be labeled and a semantic segmentation model; for each labeled sample in the automatically labeled sample set, judging whether the instances contained in the labeled sample need to be adjusted based on the confidence of at least one instance contained in the labeled sample; adjusting the instances contained in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set; and determining the labeling quality of the manually labeled sample set based on the confidence of at least one instance contained in each labeled sample in the manually labeled sample set. Compared with the prior art method in which quality inspectors review labeled samples based on subjective experience, the above method does not rely on human judgment, and can quantify the evaluation results of the labeled sample set, thus being more credible.

Description

Technical Field

[0001] The present invention relates to the field of high-precision maps, and in particular to a method and device for determining annotation quality. Background Art

[0002] Training deep learning semantic segmentation models requires a large number of labeled samples. The labeling requester typically delegates the labeling task to a labeling service provider. The labeling service provider performs the labeling task based on the client's needs, generating a large number of labeled samples and delivering them to the client. Because the quality of the labeled samples affects the recognition accuracy of the trained model, the labeling service provider must perform quality inspection on the labeled samples before delivery.

[0003] In the prior art, after obtaining a large number of labeled samples, these labeled samples are manually quality-checked in full or on a portion of the samples. However, quality inspectors may have errors in cognition and understanding, and the quality inspection results are inaccurate. Summary of the Invention

[0004] The present invention provides a method and device for determining labeling quality, which are used to improve the evaluation accuracy of labeling samples.

[0005] In a first aspect, the present invention provides a method for determining labeling quality, comprising: obtaining an automatically labeled sample set based on all samples to be labeled and a semantic segmentation model, wherein each labeled sample in the automatically labeled sample set contains at least one instance, and each instance has a corresponding confidence level; for each labeled sample in the automatically labeled sample set, judging whether it is necessary to adjust the instance contained in the labeled sample based on the confidence level of at least one instance contained in the labeled sample; adjusting the instances contained in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set; and determining the labeling quality of the manually labeled sample set based on the confidence level of at least one instance contained in the labeled samples that need to be adjusted in the automatically labeled sample set and the confidence level of at least one instance contained in each labeled sample in the manually labeled sample set.

[0006] The above method introduces the concept of confidence to quantify the quality of labeled samples. Compared with the existing technical quality inspectors who review labeled samples based on subjective experience, the above method does not rely on human judgment and can quantify the evaluation results of the labeled sample set, which is more reliable.

[0007] Optionally, the automatic labeling sample set is obtained based on all samples to be labeled and the semantic segmentation model, including: inputting all samples to be labeled into the semantic segmentation model to obtain the probability that each pixel point on each sample to be labeled belongs to various categories, and the category includes at least one of the following: people, vehicles, lane lines or traffic lights; for each sample to be labeled, according to the probability that each pixel point on the sample to be labeled belongs to various categories, the sample to be labeled is instantiated to obtain the automatic labeling sample set, wherein each instantiated sample to be labeled constitutes a labeled sample in the automatic labeling sample set.

[0008] Optionally, the sample to be labeled is instantiated according to the probability that each pixel point on the sample to be labeled belongs to various categories, including: extracting edge pixel points of all targets contained in the sample to be labeled according to the probability that each pixel point on the sample to be labeled belongs to various categories; extracting instance pixel points corresponding to all targets contained in the sample to be labeled according to the edge pixel points of all targets contained in the sample to be labeled; determining the confidence corresponding to all targets contained in the sample to be labeled according to the instance pixel points corresponding to all targets contained in the sample to be labeled; smoothing all targets contained in the sample to be labeled according to the instance pixel points corresponding to all targets contained in the sample to be labeled; thinning the edge pixel points of all targets after the smoothing process to obtain edge key points of all targets, and the number of any target contained in the sample to be labeled, the category corresponding to the target, the confidence corresponding to the target, and the edge key points of the target constitute an instance.

[0009] Optionally, the edge pixel points of all targets contained in the sample to be labeled are extracted based on the probability that each pixel point on the sample to be labeled belongs to various categories, including: based on a preset probability threshold and the probability that each pixel point on the sample to be labeled belongs to various categories, finding edge pixel points on the sample to be labeled whose probability of belonging to a first category is greater than the preset probability threshold, and determining the edge pixel points on the sample to be labeled whose probability of belonging to the first category is greater than the preset probability threshold as edge pixel points of a target.

[0010] Optionally, based on the edge pixel points of all targets contained in the sample to be labeled, instance pixel points corresponding to all targets contained in the sample to be labeled are extracted, including: for each target among all targets contained in the sample to be labeled, based on the edge pixel points of the target, obtaining the circumscribed rectangle of the target; and determining all pixel points within the circumscribed rectangle as instance pixel points corresponding to the target.

[0011] Optionally, the confidence corresponding to all targets contained in the sample to be labeled is determined based on the instance pixel points corresponding to all targets contained in the sample to be labeled, including: for each target among all targets contained in the sample to be labeled, based on the instance pixel points corresponding to the target, obtaining the probability that the instance pixel point corresponding to the target belongs to the first category; taking a weighted average of the probabilities that the instance pixel points corresponding to the target belong to the first category to obtain the confidence corresponding to the target.

[0012] Optionally, the smoothing process is performed on all targets included in the sample to be labeled based on the instance pixel points corresponding to all targets included in the sample to be labeled, including: for each target among all targets included in the sample to be labeled, dividing the target into three equal parts horizontally and vertically, and if the probability value of the instance pixel points corresponding to the target decreases from the middle to both sides in the horizontal direction, or the probability value of the instance pixel points corresponding to the target decreases from the middle to both sides in the vertical direction, then determining a smoothing point in the middle part after the three divisions; and performing smoothing process on the smoothing point.

[0013] Optionally, the method determines the labeling quality of the manually labeled sample set based on the confidence of at least one instance contained in the labeled sample that needs to be adjusted in the automatically labeled sample set and the confidence of at least one instance contained in each labeled sample in the manually labeled sample set, including: mapping the instance before adjustment and the instance after adjustment based on at least one instance contained in the labeled sample that needs to be adjusted in the automatically labeled sample set and at least one instance contained in each labeled sample in the manually labeled sample set; calculating the number of common points in the instance before adjustment and the instance after adjustment; determining the rate of change based on the number of common points; determining the confidence of the instance after adjustment based on the confidence of the instance before adjustment and the rate of change; and determining the labeling quality of the manually labeled sample set based on the confidence of the instance after adjustment.

[0014] Optionally, the calculation of the number of common points in the instance before adjustment and the instance after adjustment includes: calculating a common point threshold based on the pixel points contained in the instance before adjustment and the pixel points contained in the instance after adjustment; using the Douglas Peucker algorithm to obtain the distance between each pixel point contained in the instance before adjustment and each pixel point contained in the instance after adjustment; and determining the number of common points in the instance before adjustment and the instance after adjustment based on the common point threshold and the distance between each pixel point contained in the instance before adjustment and each pixel point contained in the instance after adjustment.

[0015] Optionally, determining the change rate based on the number of common points includes: calculating the change rate using the following formula:

[0016]

[0017] Among them, Roc Indicates the rate of change, O m Indicates the number of common points, Indicates the number of pixels contained in the adjusted instance.

[0018] Optionally, determining the confidence of the instance after adjustment based on the confidence of the instance before adjustment and the rate of change includes: calculating the confidence of the instance after adjustment using the following formula:

[0019] R oa =C i +(1-C i )R oa ×100%

[0020] Among them, R oa represents the confidence of the adjusted instance, C i Represents the confidence of the instance before adjustment, R oa Indicates the rate of change.

[0021] Optionally, determining the annotation quality of the manually annotated sample set based on the confidence of the adjusted instance includes: averaging the confidences of all adjusted instances in the manually annotated sample set, and determining the obtained average value as the annotation quality of the manually annotated sample set.

[0022] In a second aspect, the present invention provides a device for determining labeling quality, comprising: an acquisition module for acquiring an automatically labeled sample set based on all samples to be labeled and a semantic segmentation model, wherein each labeled sample in the automatically labeled sample set contains at least one instance, and each instance has a corresponding confidence level; an adjustment module for judging, for each labeled sample in the automatically labeled sample set, whether it is necessary to adjust the instance contained in the labeled sample based on the confidence level of at least one instance contained in the labeled sample; adjusting the instances contained in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set; and a determination module for determining the labeling quality of the manually labeled sample set based on the confidence level of at least one instance contained in the labeled samples that need to be adjusted in the automatically labeled sample set and the confidence level of at least one instance contained in each labeled sample in the manually labeled sample set.

[0023] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method of the first aspect when executed by a processor.

[0024] In a fourth aspect, the present invention provides a server comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to implement the method of the first aspect by executing the executable instructions.

[0025] The annotation quality determination method and device provided by the present invention introduce the concept of confidence to quantify the quality of annotated samples. Compared with the existing technology in which quality inspectors review annotated samples based on subjective experience, the above method does not rely on human judgment and can quantify the evaluation results of the annotated sample set, which is more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the evaluation method for labeled samples provided by the present invention;

[0027] Figure 2 A schematic flow chart of Example 1 of the method for evaluating labeled samples provided by the present invention;

[0028] Figure 3 Schematic diagram of the sample P0 to be labeled provided by the present invention;

[0029] Figure 4 A schematic diagram of the target object outline formed by connecting edge pixels provided by the present invention;

[0030] Figure 5 Schematic diagram of the target object provided by the present invention Figure 1 ;

[0031] Figure 6 Schematic diagram of the target object provided by the present invention Figure 2 ;

[0032] Figure 7 Schematic diagram of the target object provided by the present invention Figure 3 ;

[0033] Figure 8 A schematic diagram of the smoothing processing principle provided by the present invention;

[0034] Figure 9 This is a schematic diagram of the target object after smoothing provided by the present invention;

[0035] Figure 10 A schematic flow chart of Example 2 of the method for evaluating labeled samples provided by the present invention;

[0036] Figure 11 A schematic diagram of the structure of the evaluation device for labeled samples provided by the present invention;

[0037] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings in this application. Apparently, the described embodiments are some but not all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without any creative effort belong to the scope of protection of this application.

[0039] In this application, it should be explained that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of a single item or plural items. For example, at least one (item) of a, b, or c may represent: a alone, b alone, c alone, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b, and c, where a, b, and c may be single or multiple.

[0040] The annotator who provides the sample annotation service performs the annotation task according to the annotation requirements of the requester. After the annotation is completed, before delivering the annotated sample to the requester, it is necessary to review the annotated sample to ensure that the annotation quality meets the standard. In the prior art, the full amount of the annotated sample is reviewed manually or a part of it is selected for review. However, there are cognitive and understanding errors among the reviewers, and the review results are not accurate. Moreover, there is no quantitative standard for the review results, and the evaluation of the annotation quality is not delicate enough.

[0041] In order to achieve the goal of no longer relying on manual evaluation of labeled samples and to ensure that the evaluation results have quantitative characteristics, it is considered that a quantitative factor can be introduced to measure the accuracy of the labeling results. Since the semantic segmentation model can infer the input image and obtain the probability that each pixel on the image belongs to various categories such as people, vehicles, lane lines, etc., it is considered that the above-mentioned quantitative characteristics can be achieved based on this probability. Specifically, all samples to be labeled are first input into a pre-trained semantic segmentation model to obtain an automatically labeled sample set. Each labeled sample in the automatically labeled sample set contains at least one instance, and each instance has a corresponding confidence level. For each labeled sample in the automatically labeled sample set, the average value of the confidence level of at least one instance contained in the labeled sample can be calculated. This average value is used to represent the confidence level of the labeled sample. Then, it is determined whether the confidence level of the labeled sample reaches the preset value. If it reaches the preset value, it means that the quality of the labeled sample has met the delivery requirements and no longer requires manual adjustment. If it does not reach the preset value, it means that the quality of the labeled sample is poor and requires manual adjustment. Adjustments are made to each annotated sample in the automatically annotated sample set that requires manual adjustment, resulting in a manually annotated sample set. These adjustments may involve operations such as deleting, adding, or moving pixels within a specific instance of the annotated sample. The confidence level of the adjusted instance represents the accuracy of the manual annotation for that instance. The average confidence level of all or some of the instances in the manually annotated sample set is calculated and used as the annotation quality of the manually annotated sample set. This method does not rely on human judgment and can quantify the annotation quality of the manually annotated sample set, providing greater credibility.

[0042] It should be noted that the pre-trained semantic segmentation model mentioned above can be trained to segment all semantics, or it can be trained to segment the semantics that the demander needs to label. For example, if the demander requires labeling of people and vehicles, the semantic segmentation model can be trained to segment only people and vehicles.

[0043] The evaluation method for labeled samples provided by the present invention can be implemented by any device with corresponding processing capabilities, such as mobile phones, tablet computers, laptop computers, and desktop computers.

[0044] Figure 1 This is a schematic diagram of the principle of the evaluation method for labeled samples provided by the present invention. Figure 1The method of the present invention is explained by taking the sample to be labeled P0 as an example. All samples to be labeled can be processed using the same method. The sample to be labeled P0 is input into the semantic segmentation model, and the inference result output by the semantic segmentation model is instantiated to obtain the labeled sample P1. The average value of the confidence of at least one instance contained in the labeled sample P1 is calculated, and the average value is used to represent the confidence of the labeled sample P1. It is judged whether the confidence of the labeled sample P1 reaches the preset value. If not, the instance in the labeled sample P1 is manually adjusted to obtain the labeled sample P2. The instances in the labeled sample P1 and the labeled sample P2 are mapped according to the number. Assuming that the number is 001, the confidence of the instance 001 in the labeled sample P2 is calculated based on the instance 001 extracted from the labeled sample P1 and the instance 001 extracted from the labeled sample P2. After calculating the accuracy of all instances contained in all labeled samples in the manually labeled sample set, the accuracy of the manually labeled sample set can be obtained by taking the average value, which can be used as the evaluation result of the manually labeled sample set.

[0045] The above evaluation involves the evaluation of the sample set dimension. In addition, the accuracy of other dimensions can also be calculated to achieve evaluation of other dimensions, such as image dimension, classification dimension, labeling personnel dimension, etc., which will not be detailed in this invention.

[0046] The following describes in detail the technical solutions of the present invention and how the technical solutions of this application solve the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0047] Figure 2 This is a flow chart of Example 1 of the method for evaluating labeled samples provided by the present invention. The method for evaluating labeled samples provided in this embodiment includes:

[0048] S201. Obtain an automatically labeled sample set based on all samples to be labeled and a semantic segmentation model. Each labeled sample in the automatically labeled sample set contains at least one instance, and each instance has a corresponding confidence level.

[0049] In one possible implementation, all samples to be labeled can be input into the semantic segmentation model to obtain the probability that each pixel in each sample belongs to various categories. Then, for each sample to be labeled, the probability that each pixel in the sample belongs to various categories is instantiated to obtain an automatically labeled sample set, where each instantiated sample to be labeled constitutes a labeled sample in the automatically labeled sample set.

[0050] The instantiation method provided by the present invention is described below using the sample P0 to be labeled as an example. It is understandable that the application of the instantiation method is not limited to the sample P0 to be labeled, but is applicable to all samples to be labeled:

[0051] Step A: According to the probability that each pixel point on the sample to be labeled P0 belongs to various categories, the edge pixel points of all target objects contained in the sample to be labeled P0 are extracted.

[0052] Specifically, based on the preset probability threshold and the probability that each pixel point on the sample to be labeled P0 belongs to various categories, the edge pixel points on the sample to be labeled P0 whose probability of belonging to the first category is greater than the preset probability threshold are found, and the edge pixel points on the sample to be labeled P0 whose probability of belonging to the first category is greater than the preset probability threshold are determined as edge pixel points of a target object.

[0053] The following combination Figure 3 and Figure 4 Explanation of this step:

[0054] Assume that Figure 3 For the sample to be labeled P0, after inputting it into the semantic segmentation model, the probability of each pixel point on the sample to be labeled P0 belonging to various categories such as people, vehicles, lane lines, etc. can be obtained. For any target object in the sample to be labeled P0, the closer it is to the center of the target object, the higher the probability that the pixel point belongs to the corresponding category. In order to extract the edges of each target object contained in the sample to be labeled P0, for example, in order to extract the edges of the vehicle on the sample to be labeled P0, a probability threshold can be preset, and the edge pixels whose probability of belonging to the vehicle category is greater than the preset probability threshold can be found from the sample to be labeled P0, such as Figure 4 As shown, these edge pixels are determined as edge pixels of the target object, the vehicle.

[0055] Step B: extract instance pixel points corresponding to all target objects contained in the sample to be labeled P0 based on the edge pixel points of all target objects contained in the sample to be labeled P0.

[0056] Since the probability that an edge pixel of a target object belongs to the corresponding category cannot reflect the confidence that the entire target object belongs to the corresponding category, in order to calculate the confidence of the target object, after extracting the edge pixel of the target object, all instance pixel points corresponding to the target object must be found.

[0057] In one possible implementation, for each target object among all target objects included in the sample to be labeled P0, the bounding rectangle of the target object is obtained based on the edge pixels of the target object, and then all pixels within the bounding rectangle are determined as instance pixels corresponding to the target object.

[0058] In another possible implementation, for each target object in all target objects included in the sample to be labeled P0, the open source computer vision and machine learning software OpenCV is used to extract the instance pixel points corresponding to the target object.

[0059] Step C: Determine the confidence levels corresponding to all target objects included in the sample to be labeled P0 based on the instance pixels corresponding to all target objects included in the sample to be labeled P0.

[0060] In one possible implementation, for each target object among all target objects contained in the sample to be labeled P0, the probability that the instance pixel point corresponding to the target object belongs to the first category is obtained based on the instance pixel point corresponding to the target object; the weighted average of the probabilities that the instance pixel point corresponding to the target object belongs to the first category is calculated to obtain the confidence level corresponding to the target object.

[0061] The following combination Figure 5-Figure 7 Explanation of this step:

[0062] Assume that Figure 5 The instance pixel points corresponding to a target object extracted by the above method are very dense and cannot be Figure 5 The probability of marking all instance pixels in , in order to illustrate the principle of this step, see Figure 6 As shown, here we select Figure 5 A few points on the edge of the target object are used as an example for illustration. It can be understood by those skilled in the art that: in actual calculation, Figure 5 All instance pixels corresponding to the target object in need to participate in the calculation.

[0063] See also Figure 7 As shown, in Figure 5The probabilities of the pixels selected in the image belonging to the vehicle category are: 93, 96, 94, 96, 92, 98, 95, 94, 97, 95, 96, 94, 93, 93, 91, 96, 95, 98, 96, 94, 92, 94, 93, 92, 98, 95, 94, 93, 96, 98, 92, and 90. Taking the weighted average of these probabilities yields the confidence score for the target object. Specifically, a weight can be preset, such as 0.85. Probabilities less than 85 are multiplied by 0.85, effectively using 85 as the dividing line. Confidence scores for pixels with probabilities less than 85 are calculated as 85%. This weight is intended to minimize the impact of low-probability pixels on the confidence score of the target object. Since all of the above probabilities are greater than 85, the weighted average is essentially a geometric mean, resulting in a confidence score of 94.4456 for the target object.

[0064] Step D: Smoothing all target objects included in the sample to be labeled P0 according to the instance pixels corresponding to all target objects included in the sample to be labeled P0.

[0065] In one possible implementation, for each target object among all target objects included in the sample to be labeled P0, the target object is divided into three equal parts horizontally and vertically. If the probability value of the instance pixel point corresponding to the target object decreases from the middle to both sides in the horizontal direction, or the probability value of the instance pixel point corresponding to the target object decreases from the middle to both sides in the vertical direction, then a smoothing point is determined in the middle part after the three-part division, and smoothing is performed on the smoothing point.

[0066] The following combination Figure 8 Explanation of this step:

[0067] See also Figure 3 As shown in the figure, after extracting the edge pixels of the target object from the sample to be labeled P0, since the target object is actually composed of two target objects stuck together, it needs to be smoothed to separate it into two target objects. As described above, within a target object, the probability value should normally decrease from the center to the outside. Therefore, see Figure 8 As shown, the target object can be divided into three equal parts horizontally and then determine whether the probability values of the instance pixels gradually decrease from the center to the sides. If so, it is considered that there is no horizontal adhesion; otherwise, it is considered that there is horizontal adhesion. Similarly, the target object can be divided into three equal parts vertically and then determine whether the probability values of the instance pixels gradually decrease from the center to the sides. If so, it is considered that there is no vertical adhesion; otherwise, it is considered that there is vertical adhesion.

[0068] When there is adhesion in the horizontal direction or in the vertical direction, a smoothing point can be determined at the middle equal division of the corresponding dimension, and then smoothing is performed at the smoothing point. Figure 8 As mentioned above, the probability value of point A in the middle bisection has a sudden change compared to the pixel on its right, so it can be determined that this is not a sticking point. Similarly, the probability value of point B has a sudden change compared to the pixel on its left, so it can be determined that this is also not a sticking point. However, the probability value of point C has a smooth transition compared to the pixel on its left and right, so it can be determined that there may be sticking here. Therefore, this point can be used as a smoothing point to smooth the target object. The result after smoothing is as follows: Figure 9 shown.

[0069] For issues like jagged edges and burrs on edge pixels, existing filters and other methods can be used to address them. For the broken target object on the unlabeled sample P0, this could be caused by partial occlusion, insufficient resolution, insufficient light source, or an incorrect camera angle. For example, a light pole could be misclassified as several target objects during the edge extraction process due to occlusion by vegetation. To address this issue, target objects with similar locations and confidence levels on the unlabeled sample P0 can be aggregated into a single target object using a classifier.

[0070] Step E: Thinning the edge pixels of all target objects after smoothing to obtain edge key points of all target objects.

[0071] Since the edge pixel points extracted in step A are very dense and require high computer processing performance, the Douglas Peucker algorithm can be used to perform thinning processing on the edge pixel points of each target object extracted in step A.

[0072] After steps A and B above, the classification, confidence, and edge key points of each target object on the sample to be labeled P0 can be obtained. In addition, each target object on the sample to be labeled P0 can be numbered uniquely. In this way, the number of any target object on the sample to be labeled P0, its corresponding category, confidence, and key points constitute an instance. The confidence corresponding to the target object can be assigned to each key point corresponding to the target object.

[0073] S202 : For each labeled sample in the automatically labeled sample set, determine whether the instance included in the labeled sample needs to be adjusted according to the confidence of at least one instance included in the labeled sample.

[0074] In one possible implementation, for each labeled sample in the automatically labeled sample set, the confidence of each instance in the labeled sample is calculated by the method of S201, and then the confidence of all instances in the labeled sample is averaged to determine whether the average value reaches the preset value. If so, it means that the quality of the labeled sample has met the delivery requirements and no manual adjustment is required. If not, it means that the quality of the labeled sample is poor and requires manual adjustment.

[0075] S203: Adjust the instances included in the labeled samples that need to be adjusted in the automatic labeled sample set to obtain a manually labeled sample set.

[0076] In one possible implementation, after obtaining the automatically labeled sample set through S101, the labeler can open the labeled samples in the automatically labeled sample set through the user interface and adjust the instances in the labeled samples, which may be to add or delete pixels in the instances, or to add or delete instances.

[0077] S204: Determine the labeling quality of the manually labeled sample set according to the confidence of at least one instance included in each labeled sample in the manually labeled sample set.

[0078] Specifically, first, based on the confidence of at least one instance contained in the labeled sample that needs to be adjusted in the automatically labeled sample set, the confidence of at least one instance contained in each labeled sample in the manually labeled sample set is calculated; then, based on the confidence of at least one instance contained in each labeled sample in the manually labeled sample set, the labeling quality of the manually labeled sample set is determined.

[0079] The above evaluation involves the evaluation of the sample set dimension. In addition, the accuracy of other dimensions can also be calculated to achieve evaluation of other dimensions, such as image dimension, classification dimension, labeling personnel dimension, etc., which will not be detailed in this invention.

[0080] The labeling quality determination method provided in this embodiment quantifies the quality of labeled samples by introducing the concept of confidence. Compared with the existing technology in which quality inspectors review labeled samples based on subjective experience, the above method does not rely on human judgment and can quantify the evaluation results of the labeled sample set, which is more reliable.

[0081] Figure 10 This is a flow chart of the second embodiment of the evaluation method for labeled samples provided by the present invention. This embodiment introduces the implementation of S204. Figure 10 As shown, the evaluation method of the labeled samples provided in this embodiment includes:

[0082] S1001: Obtain an automatically labeled sample set based on all samples to be labeled and a semantic segmentation model, wherein each labeled sample in the automatically labeled sample set contains at least one instance.

[0083] S1002: For each labeled sample in the automatically labeled sample set, determine whether it is necessary to adjust the instance included in the labeled sample according to the confidence of at least one instance included in the labeled sample.

[0084] S1003: Adjust the instances included in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set.

[0085] The implementation of the above S1001-S1003 can be found in the above embodiment, and the present invention will not be repeated here. The following describes how to determine the confidence level of the instance included in each labeled sample in the manually labeled sample set, including:

[0086] S1004 : Mapping the instance before adjustment and the instance after adjustment according to at least one instance included in the labeled sample that needs to be adjusted in the automatically labeled sample set and at least one instance included in each labeled sample in the manually labeled sample set.

[0087] Take any labeled sample in the manually labeled sample set as an example. This sample is obtained after the annotator has adjusted an instance included in a labeled sample in the automatically labeled sample set. This adjustment may involve adjusting pixels within the instance or adding or deleting instances. The annotator needs to number the added instances. The instance number can be used to find the corresponding instance in the two labeled samples before and after the adjustment. Here, it is assumed that the two instances found in the two samples are the instance before and after the adjustment.

[0088] Here are some examples:

[0089] In the automatically labeled sample set, labeled sample A contains three instances numbered 001, 002, and 003. After the annotator adjusts these instances, the labeled sample A1 in the manually labeled sample set contains four instances numbered 002, 003, and 004. The instances where the mapping succeeds are 002 and 003, while the instances where the mapping fails are 001 and 004. For example, 002 in A can be the instance before the adjustment, and 002 in A1 can be the instance after the adjustment.

[0090] S1005: Calculate the number of common points in the instance before adjustment and the instance after adjustment.

[0091] In one possible implementation, first, a common point threshold is calculated based on the pixels included in the instance before adjustment and the pixels included in the instance after adjustment; then, the Douglas Peucker algorithm is used to calculate the distance between each pixel included in the instance before adjustment and each pixel included in the instance after adjustment; then, based on the common point threshold and the distance between each pixel included in the instance before adjustment and each pixel included in the instance after adjustment, the number of common points in the instance before adjustment and the instance after adjustment is determined.

[0092] Specifically, as described above, each instance contains several edge keypoints, which are stored in a certain order. The first keypoint stored in the instance before and after adjustment can be retrieved, and the distance between the two keypoints can be calculated using the Douglas Peucker algorithm. This distance is used as the common point threshold. Next, the Douglas Peucker algorithm is used to calculate the distance between each pixel in the instance before adjustment and each pixel in the instance after adjustment. If the distance between a pixel in the instance before adjustment and all pixels in the instance after adjustment is less than the common point threshold, the pixel is considered a common point. This judgment is repeated for all pixels in the instance before adjustment, and the number of common points is calculated.

[0093] S1006. Determine the rate of change based on the number of common points.

[0094] In one possible implementation, the change rate can be calculated using the following formula:

[0095]

[0096] Among them, R oc Indicates the rate of change, O m Indicates the number of common points, Indicates the number of pixels contained in the adjusted instance.

[0097] S1007. Determine the confidence level of the adjusted instance based on the rate of change.

[0098] In one possible implementation, the confidence level of the adjusted instance may be calculated using the following formula:

[0099] R oa =C i +(1-C i )R oa ×100%

[0100] Among them, R oa represents the confidence of the adjusted instance, C i Represents the confidence of the instance before adjustment, R oa Indicates the rate of change.

[0101] It should be noted that the instances contained in the two labeled samples before and after the adjustment may also fail to map, such as instance 001 in labeled sample A and instance 004 in labeled sample A1. Among them, instance 004 in labeled sample A1 does not exist in labeled sample A, indicating that it was newly added by the annotator, and its confidence can be set to 100. Instance 001 in labeled sample A does not exist in labeled sample A1, indicating that the annotator deleted it, and it can be directly discarded.

[0102] S1008: Determine the labeling quality of the manually labeled sample set according to the adjusted confidence level of the instance.

[0103] Specifically, the confidences of all adjusted instances in the manually labeled sample set are averaged, and the obtained average value is determined as the labeling quality of the manually labeled sample set.

[0104] The evaluation method for labeled samples provided in this embodiment provides a feasible way to obtain the confidence of manually labeled samples, and introduces the concept of confidence to quantify the quality of labeled samples. Compared with the existing technology in which quality inspectors review labeled samples based on subjective experience, the above method does not rely on human judgment, and can quantify the evaluation results of the manually labeled sample set, which is more reliable.

[0105] Figure 11 This is a schematic diagram of the structure of the evaluation device for labeling samples provided by the present invention. Figure 11 As shown, the evaluation device for labeled samples provided by the present invention includes:

[0106] An acquisition module 1101 is configured to acquire an automatically annotated sample set based on all samples to be annotated and a semantic segmentation model, wherein each annotated sample in the automatically annotated sample set includes at least one instance, and each instance has a corresponding confidence level.

[0107] An adjustment module 1102 is configured to determine, for each labeled sample in the automatically labeled sample set, whether an instance included in the labeled sample needs to be adjusted based on the confidence level of at least one instance included in the labeled sample; and adjust the instances included in the labeled samples in the automatically labeled sample set that need to be adjusted to obtain a manually labeled sample set.

[0108] The determination module 1103 is configured to determine the labeling quality of the manually labeled sample set according to the confidence level of at least one instance included in each labeled sample in the manually labeled sample set.

[0109] Optionally, the acquisition module 1101 is specifically configured to:

[0110] Input all samples to be labeled into the semantic segmentation model to obtain the probability that each pixel on each sample to be labeled belongs to various categories, where the categories include at least one of the following: person, vehicle, lane line, or traffic light;

[0111] For each sample to be labeled, the sample to be labeled is instantiated according to the probability that each pixel point on the sample to be labeled belongs to various categories to obtain the automatic labeling sample set, wherein each instantiated sample to be labeled constitutes a labeled sample in the automatic labeling sample set.

[0112] Optionally, the acquisition module 1101 is specifically configured to:

[0113] Extracting edge pixels of all targets contained in the sample to be labeled based on the probability that each pixel on the sample to be labeled belongs to each category;

[0114] Extracting instance pixel points corresponding to all targets contained in the sample to be labeled according to the edge pixel points of all targets contained in the sample to be labeled;

[0115] Determining the confidence levels corresponding to all the targets included in the sample to be labeled based on the instance pixels corresponding to all the targets included in the sample to be labeled;

[0116] Smoothing all objects included in the sample to be labeled according to instance pixel points corresponding to all objects included in the sample to be labeled;

[0117] The edge pixel points of all the targets after the smoothing process are thinned out to obtain the edge key points of all the targets. The number of any target included in the sample to be labeled, the classification corresponding to the target, the confidence level corresponding to the target, and the edge key points of the target constitute an instance.

[0118] Optionally, the acquisition module 1101 is specifically configured to:

[0119] According to a preset probability threshold and the probability that each pixel point on the sample to be labeled belongs to various categories, edge pixel points whose probability of belonging to the first category is greater than the preset probability threshold are found from the sample to be labeled, and the edge pixel points on the sample to be labeled whose probability of belonging to the first category is greater than the preset probability threshold are determined as edge pixel points of a target.

[0120] Optionally, the acquisition module 1101 is specifically configured to:

[0121] For each target among all targets included in the sample to be labeled, obtaining the bounding rectangle of the target based on the edge pixels of the target;

[0122] All pixel points within the circumscribed rectangle are determined as instance pixel points corresponding to the target.

[0123] Optionally, the acquisition module 1101 is specifically configured to:

[0124] For each target in all targets included in the sample to be labeled, the open source computer vision and machine learning software OpenCV is used to extract the instance pixel points corresponding to the target.

[0125] Optionally, the acquisition module 1101 is specifically configured to:

[0126] For each target among all targets included in the sample to be labeled, obtaining, based on the instance pixel points corresponding to the target, a probability that the instance pixel point corresponding to the target belongs to the first category;

[0127] A weighted average of the probabilities that the instance pixel points corresponding to the target belong to the first category is calculated to obtain a confidence level corresponding to the target.

[0128] Optionally, the acquisition module 1101 is specifically configured to:

[0129] For each target among all targets included in the sample to be labeled, divide the target into three equal parts in the horizontal and vertical directions. If the probability value of the instance pixel points corresponding to the target decreases from the middle to the two sides in the horizontal direction, or the probability value of the instance pixel points corresponding to the target decreases from the middle to the two sides in the vertical direction, then determine a smoothing point on the middle part after the three-part division;

[0130] Smoothing is performed on the smoothing points.

[0131] Optionally, the determining module 1103 is specifically configured to:

[0132] Mapping, based on at least one instance included in the labeled sample that needs to be adjusted in the automatically labeled sample set and at least one instance included in each labeled sample in the manually labeled sample set, to obtain an instance before adjustment and an instance after adjustment;

[0133] Calculating the number of common points in the instance before adjustment and the instance after adjustment;

[0134] Determine the rate of change based on the number of common points;

[0135] Determining the confidence level of the instance after adjustment according to the confidence level of the instance before adjustment and the rate of change;

[0136] The labeling quality of the manually labeled sample set is determined according to the adjusted confidence level of the instance.

[0137] Optionally, the determining module 1103 is specifically configured to:

[0138] Calculating a common point threshold according to the pixel points included in the instance before adjustment and the pixel points included in the instance after adjustment;

[0139] Using the Douglas Peucker algorithm, the distance between each pixel point included in the instance before the adjustment and each pixel point included in the instance after the adjustment is calculated;

[0140] The number of common points in the instance before adjustment and the instance after adjustment is determined according to the common point threshold and the distance between each pixel point included in the instance before adjustment and each pixel point included in the instance after adjustment.

[0141] Optionally, the determining module 1103 is specifically configured to:

[0142] The rate of change is calculated using the following formula:

[0143]

[0144] Among them, R oc Indicates the rate of change, O m Indicates the number of common points, Indicates the number of pixels contained in the adjusted instance.

[0145] Optionally, the determining module 1103 is specifically configured to:

[0146] The confidence of the adjusted instance is calculated using the following formula:

[0147] R oa =C i +(1-C i )R oa ×100%

[0148] Among them, R oa represents the confidence of the adjusted instance, C i Represents the confidence of the instance before adjustment, R oa Indicates the rate of change.

[0149] Optionally, the determining module 1103 is specifically configured to:

[0150] The confidence scores of all adjusted instances in the manually labeled sample set are averaged, and the obtained average score is determined as the labeling quality of the manually labeled sample set.

[0151] The evaluation device for labeled samples provided in this embodiment can be used to execute the steps in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be described in detail here.

[0152] Figure 12Schematic diagram of the hardware structure of the electronic device provided by the present invention. Figure 12 As shown, the electronic device of this embodiment may include:

[0153] The memory 1201 is used to store program instructions.

[0154] The processor 1202 is configured to implement the annotation quality determination method described in any of the above embodiments when the program instructions are executed. The specific implementation principles can be found in the above embodiments, which will not be described in detail in this embodiment.

[0155] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the quality of annotations described in any one of the above embodiments is implemented.

[0156] The present invention also provides a program product, which includes a computer program, which is stored in a readable storage medium. At least one processor can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device implements the annotation quality determination method described in any of the above embodiments.

[0157] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional units.

[0160] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated: ROM), a random access memory (English: Random Access Memory, abbreviated: RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0161] It should be understood that the processor described in the present invention may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in this application may be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in the processor.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining the quality of annotation, characterized in that: include: According to all samples to be labeled and the semantic segmentation model, an automatically labeled sample set is obtained, wherein each labeled sample in the automatically labeled sample set contains at least one instance, and each instance has a corresponding confidence level; For each labeled sample in the automatically labeled sample set, judging whether it is necessary to adjust the instance included in the labeled sample according to the confidence of at least one instance included in the labeled sample; Adjusting instances included in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set; The labeling quality of the manually labeled sample set is determined according to the confidence of at least one instance contained in each labeled sample in the manually labeled sample set.

2. The method according to claim 1, characterized in that The method of obtaining an automatically labeled sample set based on all samples to be labeled and the semantic segmentation model includes: Input all samples to be labeled into the semantic segmentation model to obtain the probability that each pixel on each sample to be labeled belongs to various categories, where the categories include at least one of the following: person, vehicle, lane line, or traffic light; For each sample to be labeled, the sample to be labeled is instantiated according to the probability that each pixel point on the sample to be labeled belongs to various categories to obtain the automatic labeling sample set, wherein each instantiated sample to be labeled constitutes a labeled sample in the automatic labeling sample set.

3. The method according to claim 2, characterized in that The instantiating of the sample to be labeled according to the probability that each pixel point on the sample to be labeled belongs to each category includes: Extracting edge pixels of all targets contained in the sample to be labeled based on the probability that each pixel on the sample to be labeled belongs to each category; Extracting instance pixel points corresponding to all targets contained in the sample to be labeled according to the edge pixel points of all targets contained in the sample to be labeled; Determining the confidence levels corresponding to all the targets included in the sample to be labeled based on the instance pixels corresponding to all the targets included in the sample to be labeled; Smoothing all objects included in the sample to be labeled according to instance pixel points corresponding to all objects included in the sample to be labeled; The edge pixel points of all the targets after the smoothing process are thinned out to obtain the edge key points of all the targets. The number of any target included in the sample to be labeled, the classification corresponding to the target, the confidence level corresponding to the target, and the edge key points of the target constitute an instance.

4. The method according to claim 3, characterized in that The step of extracting edge pixels of all targets contained in the sample to be labeled according to the probability that each pixel on the sample to be labeled belongs to each category includes: According to a preset probability threshold and the probability that each pixel point on the sample to be labeled belongs to various categories, edge pixel points whose probability of belonging to the first category is greater than the preset probability threshold are found from the sample to be labeled, and the edge pixel points on the sample to be labeled whose probability of belonging to the first category is greater than the preset probability threshold are determined as edge pixel points of a target.

5. The method according to claim 3, characterized in that The extracting, based on the edge pixel points of all the targets included in the sample to be labeled, instance pixel points corresponding to all the targets included in the sample to be labeled includes: For each target among all targets included in the sample to be labeled, obtaining the bounding rectangle of the target based on the edge pixels of the target; All pixel points within the circumscribed rectangle are determined as instance pixel points corresponding to the target.

6. The method according to claim 4, characterized in that The step of determining the confidence levels corresponding to all the targets included in the sample to be labeled based on the instance pixels corresponding to all the targets included in the sample to be labeled includes: For each target among all targets included in the sample to be labeled, obtaining, based on the instance pixel points corresponding to the target, a probability that the instance pixel point corresponding to the target belongs to the first category; A weighted average of the probabilities that the instance pixel points corresponding to the target belong to the first category is calculated to obtain a confidence level corresponding to the target.

7. The method according to claim 3, characterized in that The smoothing process is performed on all objects included in the sample to be labeled according to the instance pixels corresponding to all objects included in the sample to be labeled, including: For each target among all targets included in the sample to be labeled, divide the target into three equal parts in the horizontal and vertical directions. If the probability value of the instance pixel points corresponding to the target decreases from the middle to the two sides in the horizontal direction, or the probability value of the instance pixel points corresponding to the target decreases from the middle to the two sides in the vertical direction, then determine a smoothing point on the middle part after the three-part division; Smoothing is performed on the smoothing points.

8. The method according to any one of claims 1 to 7, characterized in that The determining the labeling quality of the manually labeled sample set according to the confidence of at least one instance contained in each labeled sample in the manually labeled sample set includes: Mapping, based on at least one instance included in the labeled sample that needs to be adjusted in the automatically labeled sample set and at least one instance included in each labeled sample in the manually labeled sample set, to obtain an instance before adjustment and an instance after adjustment; Calculating the number of common points in the instance before adjustment and the instance after adjustment; Determine the rate of change based on the number of common points; Determining the confidence level of the instance after adjustment according to the confidence level of the instance before adjustment and the rate of change; The labeling quality of the manually labeled sample set is determined according to the adjusted confidence level of the instance.

9. The method according to claim 8, characterized in that The calculating the number of common points between the instance before adjustment and the instance after adjustment includes: Calculating a common point threshold according to the pixel points included in the instance before adjustment and the pixel points included in the instance after adjustment; Using the Douglas Peucker algorithm, the distance between each pixel point included in the instance before the adjustment and each pixel point included in the instance after the adjustment is calculated; The number of common points in the instance before adjustment and the instance after adjustment is determined according to the common point threshold and the distance between each pixel point included in the instance before adjustment and each pixel point included in the instance after adjustment.

10. The method according to claim 8, characterized in that Determining the change rate according to the number of common points includes: The rate of change is calculated using the following formula: Among them, R oc Indicates the rate of change, O m Represents the number of common points, and N represents the number of pixels contained in the adjusted instance.

11. The method according to claim 8, characterized in that The determining the confidence level of the instance after adjustment according to the confidence level of the instance before adjustment and the rate of change includes: The confidence of the adjusted instance is calculated using the following formula: R oa =C i +(1-C i )R oa ×100% Among them, R oa represents the confidence of the adjusted instance, C i Represents the confidence of the i-th instance before adjustment, R oa Indicates the rate of change.

12. The method according to claim 11, characterized in that Determining the labeling quality of the manually labeled sample set according to the adjusted confidence level of the instance includes: The confidence scores of all adjusted instances in the manually labeled sample set are averaged, and the obtained average score is determined as the labeling quality of the manually labeled sample set.

13. A marking quality determination device, characterized in that: include: An acquisition module is used to acquire an automatically annotated sample set based on all samples to be annotated and a semantic segmentation model, wherein each annotated sample in the automatically annotated sample set contains at least one instance, and each instance has a corresponding confidence level; an adjustment module, for each labeled sample in the automatically labeled sample set, judging whether an instance included in the labeled sample needs to be adjusted based on the confidence of at least one instance included in the labeled sample; and adjusting the instances included in the labeled samples that need to be adjusted in the automatically labeled sample set to obtain a manually labeled sample set; A determination module is configured to determine the labeling quality of the manually labeled sample set based on the confidence of at least one instance included in the labeled sample that needs to be adjusted in the automatically labeled sample set and the confidence of at least one instance included in each labeled sample in the manually labeled sample set.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A server, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to implement the method according to any one of claims 1 to 12 by executing the executable instructions.

Images