A data labeling method, a labeler assessment method and a labeling result review method

Abstract

The present invention provides a data annotation method, annotator assessment method and annotation result review method, wherein the data annotation method comprises the following steps: receiving a plurality of first annotation information, wherein the first annotation information is obtained by a user annotating an annotation object according to a pre-established annotation template, wherein the pre-established annotation template comprises an association relationship between a plurality of annotation categories; and forming second annotation information of the annotation object according to the association relationship between the plurality of first annotation information and the corresponding annotation categories. By implementing the present invention, the association between annotation objects is introduced, which is suitable for complex annotation requirements, can deeply mine image information, and improves the amount of information in data annotation.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data labeling method, a labeler assessment method, and a labeling result review method. Background Art

[0002] At present, the core of supervised learning and semi-supervised learning is to process data features and provide them to machines for training. Data is the blood of artificial intelligence, and data is meaningful only when it is annotated. A large amount of high-quality training data enables the algorithm to form a better model through learning. In order to provide data with annotated information, the current annotation method is mostly crowdsourcing. There are generally two methods for crowdsourcing data annotation: offline crowdsourcing data annotation and online crowdsourcing data annotation. Regardless of the data annotation method, the annotation contents are relatively independent. For example, in the current picture, the heads and limbs of all characters are annotated separately, and there is a lack of description of the relationship between the annotation contents. As a result, when using the current annotation method for data annotation, the mined image information is relatively single and the amount of information is limited, making it difficult to further deeply understand the image information during model training. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data labeling method, a labeler assessment method and a labeling result review method to solve the defects of the prior art in data labeling that the mined image information is relatively single and the amount of information is limited.

[0004] According to a first aspect, an embodiment of the present invention provides a data labeling method, comprising the following steps: receiving multiple first labeling information, wherein the first labeling information is obtained by a user labeling a labeling object according to a pre-established labeling template, and the pre-established labeling template includes an association relationship between multiple labeling categories; forming second labeling information of the labeling object according to the multiple first labeling information and the association relationship between corresponding labeling categories.

[0005] Optionally, the method for establishing the annotation template includes: receiving multiple user-defined annotation category information and corresponding annotation method selection information, the annotation method including at least one of a border, a polygon, a key point, a multiple-choice question, and a question-and-answer question; generating a annotation template based on the multiple user-defined annotation category information and corresponding annotation method selection information.

[0006] According to the second aspect, an embodiment of the present invention provides a labeler assessment method, comprising the following steps: obtaining the assessment and labeling results of any labeler on multiple data to be labeled, the assessment and labeling results are obtained by labeling according to the data labeling method described in the first aspect or any implementation method of the first aspect; determining whether the labeler has passed the assessment based on the assessment and labeling results of the data to be labeled and the corresponding target labeling results.

[0007] Optionally, the assessment marking result and the target marking result include an assessment marking box and a marking category corresponding to the marking box, and determining whether the labeler has passed the assessment based on the assessment marking result and the corresponding target marking result of each data to be labeled includes: performing intersection and union calculation on the assessment marking box of any data to be labeled and the corresponding target marking box to obtain an intersection and union result, and obtain an accuracy evaluation result; comparing the marking category corresponding to the assessment marking box of the data to be labeled with the marking category of the corresponding target marking box to obtain a category marking evaluation result; obtaining an evaluation result of the data to be labeled based on the accuracy evaluation result and the category marking evaluation result; repeating the steps of performing intersection and union calculation on the assessment marking box of any data to be labeled and the corresponding target marking box to obtain an intersection and union result, and obtain an accuracy evaluation result, until obtaining an evaluation result of the data to be labeled based on the accuracy evaluation result and the category marking evaluation result, until obtaining an evaluation result of each data to be labeled; determining whether the labeler has passed the assessment based on the evaluation result of each data to be labeled and a preset evaluation indicator.

[0008] Optionally, the method further includes: establishing a confusion matrix according to each category labeling evaluation result, wherein the confusion matrix is ​​used to count category information of category labeling errors; and sending the data to be labeled to the labeler according to the confusion matrix.

[0009] According to the third aspect, an embodiment of the present invention provides a method for reviewing labeling results, which is applied to the auditor side and includes the following steps: receiving the labeling results of the labeler side on the data to be labeled, and the labeling results are obtained according to the data labeling method described in the first aspect or any implementation method of the first aspect; receiving the auditor's review confirmation information on the labeling results, and the review confirmation information includes review passed and review failed; when the review confirmation information is review failed, sending the labeling results to the labeler side.

[0010] Optionally, it also includes: when the review confirmation information is that the review is passed, the marking result is sent to the quality inspector end.

[0011] According to the fourth aspect, an embodiment of the present invention provides a method for reviewing annotation results, which is applied to an arbitration end and includes the following steps: receiving a complaint request initiated by an annotator end; receiving an arbitration result of the arbitrator on the complaint request, wherein the arbitration result includes at least one of an annotator error, an auditor error, and image quality not meeting target requirements; when the arbitration result of the complaint request is an auditor error, sending the arbitration result to the auditor end.

[0012] Optionally, the method further includes: when the arbitration result of the appeal request is that the labeler is wrong, sending the arbitration result to the labeler end; when the arbitration result of the appeal request is that the image quality does not meet the target requirement, deleting the labeling result.

[0013] Optionally, after receiving the appeal request initiated by the annotator for the failed annotation result review and before receiving the arbitration result of the arbitrator on the appeal request, it includes: comparing the appeal request for the failed annotation result review with historical appeal requests to determine whether the appeal request is initiated repeatedly; when the appeal request is not initiated repeatedly, displaying the appeal request.

[0014] According to the fifth aspect, an embodiment of the present invention provides a data labeling device, including: a first labeling information receiving module, used to receive multiple first labeling information, the first labeling information is obtained by a user labeling a labeling object according to a pre-established labeling template, and the pre-established labeling template includes an association relationship between multiple labeling categories; a second labeling information determination module, used to form second labeling information of the labeling object according to the multiple first labeling information and the association relationship between corresponding labeling categories.

[0015] According to the sixth aspect, an embodiment of the present invention provides a labeler assessment device, including: a first assessment and labeling result acquisition module, used to obtain the assessment and labeling results of any labeler on multiple data to be labeled, and the assessment and labeling results are obtained by labeling according to the data labeling method described in the first aspect or any implementation method of the first aspect; an assessment result judgment module, used to determine whether the labeler has passed the assessment based on the assessment and labeling results of the data to be labeled and the corresponding target labeling results.

[0016] According to the seventh aspect, an embodiment of the present invention provides a marking result review device, which is applied to the auditor side, including: a second assessment marking result acquisition module, which is used to receive the marking result of the annotator side on the data to be marked, and the marking result is obtained according to the first aspect or the data marking method of any implementation method of the first aspect; the second assessment marking result acquisition module is used to receive the auditor's review confirmation information on the marking result, and the review confirmation information includes review passed and review failed; a first sending module, which is used to send the marking result to the annotator side when the review confirmation information is review failed.

[0017] According to the eighth aspect, an embodiment of the present invention provides a labeling result review device, which is applied to an arbitration end, including: an application request receiving module, used to receive a complaint request initiated by a labeler end; an arbitration result receiving module, used to receive the arbitration result of the arbitrator on the complaint request, the arbitration result including at least one of a labeler error, a reviewer error, and image quality not meeting target requirements; a second sending module, used to send the arbitration result to the reviewer end when the arbitration result of the complaint request is a reviewer error.

[0018] According to the ninth aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data labeling method described in the first aspect or any embodiment of the first aspect, or the labeler assessment method described in the second aspect or any embodiment of the second aspect, or the labeling result review method described in the third aspect or any embodiment of the third aspect, or the fourth aspect or any embodiment of the fourth aspect are implemented.

[0019] According to the tenth aspect, an embodiment of the present invention provides a storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the data labeling method described in the first aspect or any embodiment of the first aspect, or the labeler assessment method described in the second aspect or any embodiment of the second aspect, or the labeling result review method described in the third aspect or any embodiment of the third aspect, or the fourth aspect or any embodiment of the fourth aspect.

[0020] The technical solution of the present invention has the following advantages:

[0021] The data labeling method provided in an embodiment of the present invention forms second labeling information containing the association relationship between labeling categories by associating the first labeling information according to a pre-established template. Compared with the traditional labeling method, it introduces the association between labeling objects, is suitable for complex labeling needs, can deeply mine image information, and improves the amount of information in data labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flowchart of a specific example of the data labeling method in an embodiment of the present invention;

[0024] Figure 2 This is a specific example diagram of the data labeling method in an embodiment of the present invention;

[0025] Figure 3 This is a specific example diagram of the data labeling method in an embodiment of the present invention;

[0026] Figure 4 is a flowchart of a specific example of a labeler assessment method in an embodiment of the present invention;

[0027] Figure 5 A flowchart of a specific example of a method for reviewing marking results in an embodiment of the present invention;

[0028] Figure 6 A flowchart of a specific example of a method for reviewing marking results in an embodiment of the present invention;

[0029] Figure 7 It is a principle block diagram of a specific example of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0032] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] This embodiment provides a data labeling method, such as Figure 1 As shown, the following steps are included:

[0035] S101, receiving a plurality of first annotation information, where the first annotation information is obtained by a user annotating an annotation object according to a pre-established annotation template, where the pre-established annotation template includes a plurality of annotation categories and associations between the annotation categories;

[0036] Exemplarily, the first annotation information may include position information of a border, a key point, a polygon, etc., and annotation content corresponding to the position information. Figure 2 As shown in FIG. 1 , the first annotation information may be face position information (Q-1 and Q-2) annotated in a rectangular frame or glasses position information (W-3) annotated in a rectangular frame, and other annotation contents corresponding to the face position information of Q-1 and Q-2, such as gender annotation, emotion annotation, glasses type information corresponding to the glasses position information of W-3, etc. The pre-established annotation template may be Figure 2 Take the part within the dotted line as an example, which includes Figure 3 The different annotation categories shown, as well as the hierarchical relationship between different annotation categories, include a face frame, and the gender annotation (male, female) and emotion annotation (joy, anger, sadness, happiness) corresponding to the face frame. If there is a glasses frame in the face frame, the annotation content also includes the glasses type (sunglasses, dark glasses, ordinary glasses) of the corresponding glasses frame.

[0037] The method of receiving multiple first annotation information may be receiving a user's click signal on a specific area, for example, receiving a click signal on the selection area corresponding to the male and female options, or receiving position information of borders, key points, polygons, etc. This embodiment does not limit the method of receiving multiple first annotation information, and technical personnel in this field can determine it as needed.

[0038] S102: Form second annotation information of the annotation object according to the association relationship between the plurality of first annotation information and the corresponding annotation categories.

[0039] Exemplarily, the second annotation information represents the annotation information formed by associating multiple first annotation information according to the relationship between their annotation categories. According to the association relationship between multiple first annotation information and corresponding annotation categories, the second annotation information of the annotation object can be formed by writing each first annotation information into a set according to the association relationship between the corresponding annotation categories to obtain the second annotation information of the annotation object. The expression form of the set can be:

[0040] C={c j |j∈[1,...,m]};

[0041] Among them, m represents the number of all composite annotation categories, c j = j ,S (j) >,u j Indicates the basic type of the current composite annotation category, S (j) Represents the associated subset of the current composite category, S (j) is a composite annotation type s k The set composed of is defined as follows:

[0042] S (j) ={s k |s k ∈C,k∈[1,…,l (j) ]};

[0043] Among them l (j) Represents the number of the set, when l (j) =0 is an empty set.

[0044] by Figure 3 For example,

[0045] C = {Face frame: {<Gender: Male, Female>, <Emotion: Happy, Angry, Sad, Joy>, Glasses frame:

[0046] <Glasses type: regular glasses, sunglasses, dark glasses>}}

[0047] by Figure 2 The second annotation information expression form formed by the association relationship between the multiple first annotation information corresponding to Q-1 and the corresponding annotation categories can be:

[0048] C={Face frame Q-1:{<Gender: Male>, <Emotion: Happy>, Glasses frame w-3:<

[0049] Glasses type: ordinary glasses>}}. ​

[0050] The data annotation method provided in this embodiment forms second annotation information containing the association relationship between annotation categories by associating the first annotation information according to a pre-established template. Compared with the traditional annotation method, it introduces the association between annotation objects, is suitable for complex annotation needs, can deeply mine image information, and increases the amount of information in data annotation.

[0051] As an optional implementation of this embodiment, the method of establishing the annotation template includes:

[0052] Receive multiple user-defined annotation category information and annotation method selection information, where the annotation methods include at least one of borders, polygons, key points, multiple-choice questions, and essay questions; generate an annotation template based on the multiple user-defined annotation category information and the corresponding annotation method selection information.

[0053] Exemplarily, the multiple annotation category information customized by the user may be different for different annotation tasks. For example, when annotating a person sample, the multiple annotation category information customized by the user may include face position, expression category, gender category, glasses position, glasses category, etc. When annotating a vehicle sample, the multiple annotation category information customized by the user may include vehicle category, vehicle color, etc. This embodiment does not limit the annotation category information, and those skilled in the art may determine it as needed.

[0054] by Figure 2 For example, the annotation method selection information corresponding to multiple annotation category information is described. When annotating expressions, the annotation method is multiple choice questions, and when annotating the entire face position, the annotation method is a frame. This embodiment does not limit the annotation method, and those skilled in the art can set it as needed.

[0055] The method of receiving the annotation method selection information may be to receive the selection information of a pre-built component. For example, for the annotation method in the form of a multiple-choice question, the interface stores a pre-built component representing the multiple-choice question. When the user needs the annotation method in the form of a multiple-choice question, the corresponding multiple-choice question component can be selected by dragging, pulling, or clicking on the option. The method of receiving multiple annotation category information customized by the user may be to receive the user's input information in a specific input box, for example, inputting the four options of joy, anger, sadness, and happiness in the multiple-choice question option position box. Thus, a annotation template is generated according to the multiple annotation category information customized by the user and its corresponding annotation method selection information, thereby improving the flexibility of data annotation.

[0056] This embodiment provides a method for assessing labelers. Figure 4 As shown, the following steps are included:

[0057] S201, obtaining the assessment and annotation results of a plurality of to-be-annotated data by any annotator, wherein the assessment and annotation results are obtained by annotating according to the data annotation method in the above embodiment;

[0058] Exemplarily, the multiple data to be labeled can be randomly distributed pictures or directed distributed pictures. For example, when it is necessary to assess the labeling ability of a certain type of picture of a certain labeler, the pictures of this type can be directed and sent to the labeler. This embodiment does not limit the types of the multiple data to be labeled and the distribution method, and those skilled in the art can determine it as needed. The assessment and labeling results of the multiple data to be labeled can include an assessment labeling box for the data to be labeled and the labeling category information. For the data labeling method, please refer to the corresponding part of the above embodiment, which will not be repeated here. The method of obtaining the assessment and labeling results of any labeler on multiple data to be labeled can be to obtain all the assessment and labeling results of the labeler for assessment. In order to reduce the amount of assessment data, a part of all the assessment and labeling results of the labeler can also be randomly extracted for assessment.

[0059] S202, determining whether the annotator has passed the assessment based on the assessment and annotation results of the data to be annotated and the corresponding target annotation results.

[0060] Exemplarily, the target labeling result represents the standard labeling result of the data to be labeled. According to the assessment labeling result of the data to be labeled and the corresponding target labeling result, the method of determining whether the labeler has passed the assessment can be determined only by the assessment labeling box of the data to be labeled and the assessment labeling box in the corresponding target labeling result; or it can be determined only by the labeling category information of the data to be labeled and the labeling category information in the target labeling result. When the amount of data to be labeled with errors in the labeling category exceeds a predetermined threshold, it is determined that the labeler has failed the assessment. This embodiment does not limit the method of determining whether the labeler has passed the assessment based on the assessment labeling result of the data to be labeled and the corresponding target labeling result, and those skilled in the art can determine it as needed.

[0061] This embodiment is described by taking the determination only through the assessment marking box of the to-be-annotated data and the assessment marking box in the corresponding target marking result as an example, and the specific method is as follows:

[0062]

[0063] Among them, R1 is the assessment annotation box, and R2 is the target annotation box in the target annotation result.

[0064] According to the scoring criteria in Table 1 and combined with the above IoU, the annotation results of the annotators are scored.

[0065] Table 1

[0066] IoU value 0 (0，0.3) [0.3，0.6) [0.6，0.8) [0.8，0.9) [0.9，1) 1.0 Score 0 30 60 70 80 90 100

[0067] When a data to be labeled contains multiple targets that need to be labeled, the score of each assessment labeling box marked by the labeler needs to be recorded. The score of each data to be labeled can be the average score of all the assessment labeling boxes in the data to be labeled to increase the difficulty of the assessment, or it can be the lowest score of all the assessment labeling boxes. This embodiment does not limit this, and technicians in this field can determine it as needed. When the score of the data to be labeled is the lowest score of all the assessment labeling boxes, its expression is expressed as: s = min{s i |i∈[1,...,t]}, where t is the number of objects that need to be labeled in the image, s i Represents the score of the i-th assessment annotation box.

[0068] By setting a passing score for each data to be annotated, it is determined whether each data to be annotated is qualified. When the percentage of qualified data to be annotated in all the data to be annotated being assessed is greater than or equal to the preset percentage, the annotator is determined to have passed the assessment; when the percentage of qualified data to be annotated in all the data to be annotated being assessed is less than the set percentage, the annotator is determined to have failed the assessment. Among them, the passing score can be set to 90 points and the preset percentage to 90%. This embodiment does not limit the passing score and the preset percentage, and those skilled in the art can set them as needed.

[0069] As an optional implementation of this embodiment, the assessment labeling result and the target labeling result include an assessment labeling box and a labeling category corresponding to the labeling box. According to the assessment labeling result of each to-be-labeled data and the corresponding target labeling result, determining whether the labeler has passed the assessment includes:

[0070] First, the intersection and union ratio of any assessment annotation frame of the data to be annotated is calculated with the corresponding target annotation frame to obtain the intersection and union ratio result and the accuracy evaluation result;

[0071] Exemplarily, the accuracy evaluation result is used to characterize whether the assessment annotation box of the annotator meets the accuracy requirement, and the accuracy evaluation result can be characterized as pass or fail. The intersection and union ratio calculation of the assessment annotation box of any data to be annotated and the corresponding target annotation box is performed, and the method of obtaining the intersection and union ratio result can refer to formula (1). When there is only one target to be annotated in a data to be annotated, the intersection and union ratio score of the annotation box in the data to be annotated can be obtained according to the scoring criteria in Table 1. When a data to be annotated contains multiple targets to be annotated, the score of each assessment annotation box in the data to be annotated is obtained according to formula (1) and Table 1, and the intersection and union ratio score of the annotation box in the data to be annotated is obtained by selecting the average value or minimum value of the assessment annotation box in the data to be annotated. When the intersection and union ratio score is greater than the preset score, the accuracy evaluation result is passed, and when the intersection and union ratio score is less than or equal to the preset score, the accuracy evaluation result is failed.

[0072] Secondly, the annotation category corresponding to the assessment annotation box is compared with the annotation category of the corresponding target annotation box to obtain the category annotation evaluation result;

[0073] Exemplarily, when there are multiple targets that need to be labeled in the data to be labeled, it is determined whether the labeling category corresponding to each assessment labeling box is consistent with the labeling category of the target labeling box. When the labeling category corresponding to any assessment labeling box is inconsistent with the labeling category of the target labeling box, the obtained category labeling evaluation result is unqualified.

[0074] Then, based on the accuracy evaluation results and the category labeling evaluation results, the evaluation results of the data to be labeled are obtained;

[0075] Exemplarily, based on the accuracy evaluation results and the category labeling evaluation results, the way to obtain the evaluation results of the data to be labeled can be that when both the accuracy evaluation results and the category labeling evaluation results are passing, the evaluation result of the data to be labeled is passing, expressed as 1; when any one of the evaluation results is failing, the evaluation result of the data to be labeled is failing, expressed as 0.

[0076] Again, the intersection and union ratio calculation is repeated for any assessment annotation frame of the data to be labeled and the corresponding target annotation frame to obtain the intersection and union ratio result, and the accuracy evaluation result is obtained. The evaluation result of the data to be labeled is obtained according to the accuracy evaluation result and the category labeling evaluation result, until the evaluation result of each data to be labeled is obtained;

[0077] Then, based on the evaluation results of each piece of data to be labeled and the preset evaluation indicators, it is determined whether the labeler has passed the assessment.

[0078] For example, the preset evaluation index may be that the pass rate in the evaluation results of the data to be labeled is higher than a preset threshold, or the number of pass results in the evaluation results of the data to be labeled is greater than a preset number, and its expression may be:

[0079]

[0080] Among them, I k Indicates that the current annotation is the kth data to be annotated, f(I k ) represents the evaluation result (0 or 1) of the kth data to be labeled, x represents the total number of all assessed data to be labeled, and y represents the preset number.

[0081] As an optional implementation of this embodiment, the labeler assessment method further includes:

[0082] According to the evaluation results of each category labeling, a confusion matrix is ​​established. The confusion matrix is ​​used to count the number of category labeling errors. According to the confusion matrix, the data to be labeled is sent to the labeler.

[0083] For example, a confusion matrix is ​​established for each piece of data to be labeled, and the number of incorrectly labeled categories in each piece of data to be labeled is counted. The specific establishment method can be expressed by the following formula:

[0084] M={m i,j |i,j∈[1,...,n]};

[0085] Among them, m i,j It indicates the number of people who are labeled as the i-th category as the j-th category. When i=j, m i,j It represents the number of annotation categories corresponding to the assessment annotation box and the target annotation box, and n is the total number of categories.

[0086] According to the confusion matrix, the way to send the data to be labeled to the labeler can be to choose m i,j The data type to be labeled corresponding to the data with the largest value in the middle is selected, and the data to be labeled of the same or similar type is selected from the system and sent to the corresponding annotator to continuously strengthen the annotator's memory and understanding and improve the training effect.

[0087] For example, a data to be labeled contains 5 types of annotations and 6 targets to be labeled. The resulting confusion matrix is ​​as follows:

[0088]

[0089] m 11 is equal to 1, indicating that the number of correctly labeled type 1 is 1, m 22 is equal to 1, indicating that the number of correctly labeled second type is 1, m 53is equal to 2, which means that the number of annotation types of the fifth category is marked as the third category, m 54 is equal to 1, indicating that the number of the 5th type of annotation is marked as 1 for the 4th type of annotation, m 55 Equal to 1, indicating that the number of correctly labeled type 5 is 1. It can be seen that the type 5 and the type 3 are the most easily confused, and the images of the most easily confused type can be given priority to the annotators.

[0090] This embodiment proposes a training and assessment process based on confusion matrix evaluation. The assessment process is evaluated through the confusion matrix, and subsequent image distribution rules are guided according to the error situation, thereby enhancing the labeling personnel's understanding of error-prone rules.

[0091] This embodiment provides a method for reviewing a marking result, which is applied to a reviewer. Figure 5 As shown, the following steps are included:

[0092] S301, receiving the labeling result of the labeler on the data to be labeled, wherein the labeling result is obtained according to the above data labeling method; for the specific content, please refer to the corresponding part of the above data labeling method, which will not be repeated here.

[0093] S302, receiving the auditor's audit confirmation information on the marking results, the audit confirmation information includes audit passed and audit failed; the method of receiving the auditor's audit confirmation information on the marking results can be receiving the auditor's input information in the target input box or receiving the auditor's selection information in the target selection area. For example, two buttons, Audit Passed and Audit Failed, are displayed on the interface. When a click signal is received on any of the buttons, it is considered that the auditor's audit confirmation information on the marking results is received.

[0094] S303, when the review confirmation information indicates that the review has not been passed, the annotation result is sent to the annotator to enable the annotator to revise it.

[0095] As an optional implementation of this embodiment, the method for reviewing the labeling results further includes: when the review confirmation information is that the review is passed, the labeling results are sent to the quality inspector end. This allows the quality inspector to conduct quality inspection on the review results. When the quality inspector conducts a sampling inspection, if the sampling inspection is qualified, the labeling results are exported; if not, the unqualified labeling results are fed back to the corresponding labeler for re-labeling.

[0096] This embodiment provides a method for reviewing a marking result, which is applied to an arbitration end, such as Figure 6 As shown, the following steps are included:

[0097] S401, receiving a complaint request initiated by the annotator;

[0098] Exemplarily, the content of the appeal request may be a complaint against an auditor's audit error, or may be a behavioral complaint initiated when the auditor refuses to audit. This embodiment does not limit the specific content of the appeal request, and those skilled in the art may determine it as needed.

[0099] S402, receiving an arbitration result of the arbitrator on the appeal request, where the arbitration result includes at least one of an error of the annotator, an error of the auditor, and the image quality not meeting the target requirement;

[0100] S403, when the arbitration result of the appeal request is an auditor error, the arbitration result is sent to the auditor end.

[0101] As an optional implementation of this embodiment, the marking result review method further includes:

[0102] When the arbitration result of the appeal request is that the annotator is wrong, the arbitration result is sent to the annotator;

[0103] When the arbitration result of the appeal request is that the image quality does not meet the target requirements, the marking result will be deleted.

[0104] As an optional implementation of this embodiment, after receiving the appeal request initiated by the annotator for the unsuccessful review of the annotation results, and before receiving the arbitration result of the arbitrator on the appeal request, the method includes:

[0105] The appeal request that fails the marking result review will be compared with the historical appeal requests to determine whether the appeal request is initiated repeatedly; when the appeal request is not initiated repeatedly, the appeal request will be displayed.

[0106] Exemplarily, the historical appeal request may include the IP address of the initiator of the appeal, the specific content of the appeal, etc. The way to compare the appeal request that failed the annotation result review with the historical appeal request can be to compare whether the appeal content of the current appeal request exists in the historical appeal request. If it exists in the historical appeal request, check whether the IP address of the initiator of the appeal is the same. If the IP addresses are the same, it can be considered that it is a repeated appeal request. For repeated appeal requests, a notification of rejection of the appeal is sent directly to the annotator. When the appeal request is not a repeated initiation, the appeal request is displayed on the arbitration end so that the arbitrator can arbitrate it.

[0107] It can be found that for images rejected by the auditor, the annotator can file an appeal, thus restricting the auditor from arbitrarily rejecting the images; for images approved by the auditor, sampling inspection is also required. Once the pass rate is too low, all images reviewed by the auditor will be sent back for re-examination. In order to arbitrate the appeal and conduct sampling inspection of images, the perspective of quality inspectors is introduced. Quality inspectors can be managers at the beginning, and then people with high evaluation indicators can be selected from annotators and auditors. When performing quality inspection operations, they need to avoid images that they have annotated or reviewed.

[0108] Through the above methods, a mechanism of mutual restraint and iteration between annotators and reviewers has been established. Annotators need to strengthen their understanding of the rules and mark drawings carefully and meticulously, and reviewers also need to review drawings impartially and objectively and not make arbitrary manipulations.

[0109] The following evaluation indicators are established for labelers:

[0110] First review pass rate Where N acc_first Indicates the number of images that the auditor passed on the first review, N checked Indicates the total number of images that the reviewer has reviewed. The higher the first review pass rate of the annotator, the faster the annotator understands the rules and the more careful and meticulous the work is. If the first review pass rate is too low, it is necessary to re-evaluate whether the annotator is qualified for the position.

[0111] Complaint initiation rate Where N protest Indicates the number of images for which the complaint was initiated, N rejected Indicates the number of images rejected by the review. The appeal initiation rate indicates the probability that the annotator uses the appeal function. If the appeal initiation rate is too high, it indicates that the annotator is suspected of abusing the appeal function.

[0112] The following evaluation indicators are established for auditors:

[0113] Audit pass rate Where N passed Indicates the number of images approved by the auditor, N checked Indicates the total number of images reviewed. If the review pass rate is too high or too low, there may be problems and need attention.

[0114] Complaint rate Where N protested Indicates the number of images for which the reviewer has been appealed, N rejected Indicates the number of images rejected by the auditor. If the appeal rate is too high, it means that there are obvious problems with the auditor's review, and it is necessary to focus on evaluating whether the auditor is competent for the position.

[0115] Sampling pass rate Where N passed_again It indicates the number of images that are also judged as passed during the sampling inspection, μ is the sampling ratio coefficient, N passed The total number of images that have been approved by the auditor. The sampling pass rate must be greater than the given threshold to indicate that the auditor's judgment result is credible.

[0116] According to the above evaluation indicators of labelers and auditors, the job competence of personnel can be evaluated in time, and the labeled data can be adjusted or re-labeled in time. Due to the existence of the mutual restraint mechanism, labelers and auditors must work carefully and meticulously. After several rounds of iterations, the retained personnel can ensure high-quality labeled data output.

[0117] This embodiment uses a data annotation device, including: a first annotation information receiving module, which is used to receive multiple first annotation information, where the first annotation information is obtained by a user annotating an annotation object according to a pre-established annotation template, and the pre-established annotation template includes an association relationship between multiple annotation categories; a second annotation information determination module, which is used to form second annotation information of the annotation object according to the association relationship between the multiple first annotation information and the corresponding annotation categories. For details, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0118] This embodiment provides a labeler assessment device, including:

[0119] The first assessment and annotation result acquisition module is used to obtain the assessment and annotation results of any annotator on multiple data to be annotated, and the assessment and annotation results are obtained by annotating according to the data annotation method described in the first aspect or any implementation method of the first aspect; the specific content refers to the corresponding part of the method embodiment, which will not be repeated here.

[0120] The assessment result judgment module is used to determine whether the labeler has passed the assessment according to the assessment labeling result of the data to be labeled and the corresponding target labeling result. For details, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0121] This embodiment provides a marking result review device, which is applied to the reviewer end, including:

[0122] The second assessment and annotation result acquisition module is used to receive the annotation results of the annotator on the data to be annotated, and the annotation results are obtained according to the data annotation method described in the first aspect or any implementation method of the first aspect; the specific content refers to the corresponding part of the method embodiment, which will not be repeated here.

[0123] The second assessment and marking result acquisition module is used to receive the auditor's review confirmation information on the marking results, and the review confirmation information includes review passed and review failed; the specific content refers to the corresponding part of the method embodiment, which will not be repeated here.

[0124] The first sending module is used to send the annotation result to the annotator terminal when the audit confirmation information indicates that the audit is not passed. For details, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0125] This embodiment provides a marking result review device, which is applied to the arbitration end and includes:

[0126] The application request receiving module is used to receive the appeal request initiated by the labeler. For details, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0127] An arbitration result receiving module is used to receive the arbitration result of the arbitrator on the appeal request, and the arbitration result includes at least one of annotator error, auditor error, and image quality not meeting target requirements; for specific content, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0128] The second sending module is used to send the arbitration result to the auditor when the arbitration result of the appeal request is an auditor error. For details, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0129] The present application also provides an electronic device, such as Figure 7 As shown, a processor 310 and a memory 320, wherein the processor 310 and the memory 320 may be connected via a bus or other means.

[0130] The processor 310 may be a central processing unit (CPU). The processor 310 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0131] The memory 320, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the data labeling method or the labeler assessment method or the labeling result review method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0132] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The one or more modules are stored in the memory 320, and when executed by the processor 310, the data labeling method or the labeler assessment method or the labeling result review method in the embodiment shown in the figure is executed.

[0134] The specific details of the above electronic device can be understood by referring to the corresponding related descriptions and effects in the embodiments shown in the figures, and will not be repeated here.

[0135] This embodiment also provides a computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the data labeling method or the labeler assessment method or the labeling result review method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0136] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A data labeling method, characterized in that: The steps include: Receive a plurality of first annotation information, where the first annotation information is obtained by a user annotating an annotation object according to a pre-established annotation template, the annotation object includes a picture, and the pre-established annotation template includes associations between a plurality of annotation categories; Each first annotation information is written into a set according to the association relationship between the corresponding annotation categories to obtain the second annotation information of the annotation object; wherein the expression form of the set is: C={c j |j∈[1,...,m]}; Among them, m represents the number of all composite annotation categories, c j = j ,S (j) >,u j Indicates the basic type of the current composite annotation category, S (j) Represents the associated subset of the current composite category, S (j) is a composite annotation type s k The set composed of is defined as follows:​ S (j) ={s k |s k ∈C,k∈[1,...,l (j) ]}; Among them l (j) Represents the number of the set, when l (j) =0 is an empty set.

2. The method according to claim 1, characterized in that The method of establishing the annotation template includes: Receive multiple user-defined annotation category information and corresponding annotation method selection information, wherein the annotation method includes at least one of a border, a polygon, a key point, a multiple-choice question, and a question-and-answer question; A labeling template is generated based on multiple labeling category information and corresponding labeling method selection information defined by the user.

3. A method for assessing annotators, characterized in that: The steps include: Obtain the assessment and annotation results of any annotator on multiple data to be annotated, the assessment and annotation results The data annotation method according to claim 1 or 2 is used for annotation, wherein the assessment annotation result and the target annotation result include an assessment annotation box and an annotation category corresponding to the annotation box; Calculate the intersection and union ratio of any assessment annotation frame of the data to be annotated and the corresponding target annotation frame to obtain the intersection and union ratio result and the accuracy evaluation result; Compare the annotation category corresponding to the assessment annotation frame of the data to be annotated with the annotation category of the corresponding target annotation frame to obtain a category annotation evaluation result; Obtaining an evaluation result of the data to be labeled according to the accuracy evaluation result and the category labeling evaluation result; Repeat the calculation of intersection and union ratio of the assessment annotation frame of any data to be labeled with the corresponding target annotation frame to obtain the intersection and union ratio result, and obtain the accuracy evaluation result, and obtain the evaluation result of the data to be labeled according to the accuracy evaluation result and the category labeling evaluation result, until the evaluation result of each data to be labeled is obtained; Determine whether the annotator has passed the assessment based on the evaluation results of each piece of data to be annotated and the preset evaluation indicators; According to each category labeling evaluation result, a confusion matrix is ​​established, wherein the confusion matrix is ​​used to count category information of category labeling errors; According to the confusion matrix, the data to be labeled is sent to the labeler.

4. A method for reviewing labeling results, characterized in that: Applied to the auditor side, including the following steps: Receive the annotation results of the annotator on the data to be annotated, the annotation results The data is annotated according to the data annotation method according to claim 1 or 2; Receiving the auditor's audit confirmation information on the marking results, wherein the audit confirmation information includes whether the audit is passed or not; When the review confirmation information indicates that the review is not passed, the annotation result is sent to the annotator terminal; When the audit confirmation information indicates that the audit is passed, the marking result is sent to the quality inspector.

5. A data labeling device, characterized in that: include: A first annotation information receiving module, configured to receive a plurality of first annotation information, wherein the first annotation information is obtained by a user annotating an annotation object according to a pre-established annotation template, the annotation object includes a picture, and the pre-established annotation template includes associations between a plurality of annotation categories; The second annotation information determination module is used to write each first annotation information into a set according to the association relationship between the corresponding annotation categories to obtain the second annotation information of the annotation object; wherein the expression form of the set is: C={c j |j∈[1,...,m]}; Among them, m represents the number of all composite annotation categories, c j = j ,S (j) >,u j Indicates the basic type of the current composite annotation category, S (j) Represents the associated subset of the current composite category, S (j) is a composite annotation type s k The set composed of is defined as follows:​ S (j) ={s k |s k ∈C,k∈[1,…,l (j) ]}; Among them l (j) Represents the number of the set, when l (j) =0 is an empty set.

6. A labeler assessment device, characterized in that: include: The first assessment and annotation result acquisition module is used to obtain the assessment and annotation results of any annotator on multiple data to be annotated. The data annotation method according to claim 1 or 2 is used for annotation, wherein the assessment annotation result and the target annotation result include an assessment annotation box and an annotation category corresponding to the annotation box; The assessment result judgment module is used to calculate the intersection and union ratio of any assessment annotation frame of the data to be annotated with the corresponding target annotation frame, obtain the intersection and union ratio result, and obtain the accuracy evaluation result; Compare the annotation category corresponding to the assessment annotation frame of the data to be annotated with the annotation category of the corresponding target annotation frame to obtain a category annotation evaluation result; Obtaining an evaluation result of the data to be labeled according to the accuracy evaluation result and the category labeling evaluation result; Repeat the calculation of intersection and union ratio of the assessment annotation frame of any data to be labeled with the corresponding target annotation frame to obtain the intersection and union ratio result, and obtain the accuracy evaluation result, and obtain the evaluation result of the data to be labeled according to the accuracy evaluation result and the category labeling evaluation result, until the evaluation result of each data to be labeled is obtained; Determine whether the annotator has passed the assessment based on the evaluation results of each piece of data to be annotated and the preset evaluation indicators; According to each category labeling evaluation result, a confusion matrix is ​​established, wherein the confusion matrix is ​​used to count category information of category labeling errors; According to the confusion matrix, the data to be labeled is sent to the labeler.

7. A marking result review device, characterized in that: Applied to the auditor side, including: A second assessment labeling result acquisition module is used to receive the labeling result of the data to be labeled by the labeler, wherein the labeling result is obtained by labeling the data labeling method according to claim 1 or 2; A second assessment and marking result acquisition module is used to receive the auditor's review confirmation information on the marking result, and the review confirmation information includes whether the review is passed or not; The first sending module is used to send the marking result to the labeler end when the review confirmation information is that the review is not passed; when the review confirmation information is that the review is passed, the marking result is sent to the quality inspector end.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data labeling method according to claim 1 or 2, the labeler assessment method according to claim 3, or the labeling result review method according to claim 4 are implemented.

9. A storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the data labeling method described in claim 1 or 2, the labeler assessment method described in claim 3, or the labeling result review method described in claim 4 are implemented.

Images