A pipelined data labeling method

Abstract

The application provides a kind of pipelined data labeling method, to solve the problem of low efficiency in prior art for complex labeling task labeling. A kind of pipelined data labeling method, comprising: S100, obtaining data to be labeled, and determining labeling scheme;S200, the overall labeling task in labeling scheme is split into several labeling subtasks;S300, for each labeling subtask, according to the preset labeling mode, select into the preset labeling mode;S400, AI pre-audit is carried out to the pre-labeling result data circulated to AI pre-audit step;S500, the data that passes the audit in all labeling subtasks is merged to form the final labeling result and output. The application can significantly reduce the complexity of the overall labeling task by decomposing the overall labeling task into several labeling subtasks, so that it is converted into a simple labeling subtask, which is convenient for artificial and AI learning and identification, and can improve the efficiency and quality of data labeling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a pipeline type data labeling method. BACKGROUND

[0002] With the advent of the big data era, the data volume of numerous industries grows exponentially. In order to better utilize the massive data, it is usually necessary to label the data so that the data can better serve various activities such as production, operation and life. At present, when labeling data, the artificial labeling method is often used. Due to the limited personal knowledge and ability, it is difficult to complete the task of labeling various data, especially when labeling complex labeling tasks, there is usually a problem of low labeling efficiency. SUMMARY

[0003] The present application provides a pipeline type data labeling method, which aims to solve the problem of low labeling efficiency when labeling complex labeling tasks in the prior art.

[0004] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0005] A pipeline type data labeling method, comprising:

[0006] S100, obtaining data to be labeled and determining a labeling scheme;

[0007] S200, splitting the overall labeling task in the labeling scheme into a plurality of labeling subtasks;

[0008] S300, for each labeling subtask, selecting a preset labeling mode according to the preset labeling mode; if the labeling mode is full labeling, all the data to be labeled in the labeling subtask is transferred to artificial labeling;

[0009] If the labeling mode is extraction labeling, all the data to be labeled in the labeling subtask is transferred to an AI pre-labeling step to obtain AI pre-labeled data and AI pre-labeled data confidence. It is judged whether each AI pre-labeled data meets the condition of directly entering the AI pre-audit step according to the AI pre-labeled data confidence. If yes, it directly enters the AI pre-audit step. If not, it is transferred to artificial labeling;

[0010] The artificial labeling data obtained after artificial labeling is transferred to the AI pre-audit step;

[0011] S400, AI pre-auditing the pre-annotation result data transferred to the AI pre-auditing step to obtain AI pre-auditing data and AI pre-auditing data confidence; the pre-annotation result data includes artificial annotation data and AI pre-annotation data judged to meet the condition of directly entering the AI pre-auditing step; the AI pre-auditing data includes pre-annotation result data audited as incorrect and pre-annotation result data audited as correct; the pre-annotation result data audited as incorrect is transferred to manual auditing;

[0012] The pre-annotation result data audited as correct is selected to enter a preset auditing mode according to a preset auditing mode; if the auditing mode is a full auditing mode, all pre-annotation result data audited as correct is transferred to manual auditing; if the auditing mode is a sampling auditing mode, the pre-annotation result data audited as correct is sorted from low to high according to the AI pre-auditing data confidence, and a certain proportion of pre-annotation result data audited as correct at the top of the sorting is sampled and transferred to manual auditing, and the pre-annotation result data audited as correct at the bottom of the sorting is directly passed through auditing;

[0013] The pre-annotation result data judged as incorrect after manual auditing is returned to the artificial annotation in step S300, and the pre-annotation result data judged as correct is passed through auditing;

[0014] S500, merging all the data audited in the annotation sub-tasks to form the final annotation result and output.

[0015] Further scheme: in the AI pre-annotation step, an AI pre-annotation model is used to pre-annotate all the to-be-annotated data in the annotation sub-tasks.

[0016] Further scheme: the AI pre-annotation model includes AI pre-annotation model one, AI pre-annotation model two and AI pre-annotation model three; the AI pre-annotation step includes:

[0017] All the to-be-annotated data in the annotation sub-tasks are respectively transferred to the AI pre-annotation model one and the AI pre-annotation model two, AI pre-annotation data one and AI pre-annotation data confidence one are obtained through the AI pre-annotation model one, and AI pre-annotation data two and AI pre-annotation data confidence two are obtained through the AI pre-annotation model two;

[0018] In step S300, whether each AI pre-annotation data meets the condition of directly entering the AI pre-auditing step is judged according to the AI pre-annotation data confidence, which includes: comparing the AI pre-annotation data confidence one and the AI pre-annotation data confidence two with the first threshold value respectively;

[0019] If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both less than the first threshold, the piece of to-be-annotated data in the annotation subtask is transferred to manual annotation;

[0020] If the AI pre-annotation data confidence one is greater than or equal to the first threshold and the AI pre-annotation data confidence two is less than the first threshold, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-auditing step;

[0021] If the AI pre-annotation data confidence two is greater than or equal to the first threshold and the AI pre-annotation data confidence one is less than the first threshold, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-auditing step;

[0022] If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both greater than or equal to the first threshold, it is judged whether the AI pre-annotation data one and the AI pre-annotation data two are the same. If they are the same, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-auditing step. If they are not the same, the AI pre-annotation model three is introduced, the AI pre-annotation data three and the AI pre-annotation data confidence three are obtained through the AI pre-annotation model three, the AI pre-annotation data confidence three is compared with the first threshold, if the AI pre-annotation data confidence three is greater than or equal to the first threshold, it is judged whether the AI pre-annotation data three is the same as the AI pre-annotation data one or the AI pre-annotation data two. If the AI pre-annotation data three is the same as the AI pre-annotation data one, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-auditing step. If the AI pre-annotation data three is the same as the AI pre-annotation data two, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-auditing step. If the AI pre-annotation data three is not the same as the AI pre-annotation data one and the AI pre-annotation data two, the piece of to-be-annotated data in the annotation subtask is transferred to manual annotation. If the AI pre-annotation data confidence three is less than the first threshold, the piece of to-be-annotated data in the annotation subtask is transferred to manual annotation;

[0023] The manual annotation data obtained after manual annotation is transferred to the AI pre-auditing step.

[0024] Based on the above technical solution: by transferring the to-be-annotated data to multiple AI pre-annotation models, the output AI pre-annotation data is not dependent on the result of one AI pre-annotation model, but the pre-annotation result data is obtained by comprehensively considering multiple AI pre-annotation models, the pre-annotation result data confidence is high, and the accuracy is high. In addition, the pre-annotation result data is obtained by comprehensively considering multiple AI pre-annotation models, and the pre-annotation result will not be batched due to the deviation of a single AI pre-annotation model.

[0025] Further scheme: AI pre-annotation data confidence one = AI pre-annotation model one original confidence * AI pre-annotation model one accuracy; AI pre-annotation data confidence two = AI pre-annotation model two original confidence * AI pre-annotation model two accuracy; AI pre-annotation data confidence three = AI pre-annotation model three original confidence * AI pre-annotation model three accuracy.

[0026] Based on the above technical scheme: when AI pre-annotation data is obtained by comprehensively using multiple AI pre-annotation models, the AI pre-annotation model accuracy is used as a weighting coefficient to increase the weight of the AI pre-annotation model with high accuracy, so that the AI pre-annotation data obtained by comprehensive use is more accurate.

[0027] Further scheme: the AI pre-annotation model includes AI pre-annotation model one and AI pre-annotation model two; the AI pre-annotation step includes:

[0028] All the to-be-labeled data in the labeling subtask are respectively forwarded to the AI pre-annotation model one and the AI pre-annotation model two, a plurality of pre-annotation data and a plurality of pre-annotation data confidence are obtained by using the AI pre-annotation model one, and a plurality of pre-annotation data and a plurality of pre-annotation data confidence are obtained by using the AI pre-annotation model two.

[0029] The pre-annotation data confidence obtained by using the AI pre-annotation model one and the pre-annotation data confidence obtained by using the AI pre-annotation model two are averaged for each pre-annotation data, the pre-annotation data with the highest pre-annotation data confidence is taken as the pre-annotation result data, and the pre-annotation result data is forwarded to the AI pre-audit step.

[0030] Based on the above technical scheme: using AI instead of manual labeling can improve the labeling efficiency; and the pre-annotation result data obtained by comprehensively using two AI pre-annotation models has high pre-annotation data confidence and high accuracy compared with the pre-annotation result data obtained by using a single AI pre-annotation model.

[0031] Further scheme: in the AI pre-audit step, an AI pre-audit model is used to perform AI pre-audit on the pre-annotation result data forwarded to the AI pre-audit step.

[0032] Further scheme: the AI pre-audit model includes AI pre-audit model one, AI pre-audit model two and AI pre-audit model three; the AI pre-audit step includes:

[0033] S401, the pre-annotation result data is respectively forwarded to the AI pre-audit model one and the AI pre-audit model two, the AI pre-audit data one and the AI pre-audit data confidence one are obtained by using the AI pre-audit model one, the AI pre-audit data two and the AI pre-audit data confidence two of the AI pre-audit model two are obtained by using the AI pre-audit model two.

[0034] S402, compare the AI pre-audit data confidence one and the AI pre-audit data confidence two with the second threshold value respectively;

[0035] If the AI pre-audit data confidence one is greater than or equal to the second threshold value and the AI pre-audit data confidence two is less than the second threshold value, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence one is taken as the AI pre-audit data confidence;

[0036] If the AI pre-audit data confidence two is greater than or equal to the second threshold value and the AI pre-audit data confidence one is less than the second threshold value, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence two is taken as the AI pre-audit data confidence;

[0037] If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both greater than or equal to the second threshold value, it is judged whether the AI pre-audit data one and the AI pre-audit data two are the same, if the same, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence of the higher one of the AI pre-audit model one and the AI pre-audit model two is taken as the AI pre-audit data confidence; if not the same, the AI pre-audit model three is introduced, the AI pre-audit data three and the AI pre-audit data confidence three are obtained through the AI pre-audit model three, the AI pre-audit data confidence three is compared with the second threshold value, if the AI pre-audit data confidence three is greater than or equal to the second threshold value, it is judged whether the AI pre-audit data three is the same as the AI pre-audit data one or the AI pre-audit data two, if the AI pre-audit data three is the same as the AI pre-audit data one, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence of the higher one of the AI pre-audit model one and the AI pre-audit model three is taken as the AI pre-audit data confidence; if the AI pre-audit data three is the same as the AI pre-audit data two, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence of the higher one of the AI pre-audit model two and the AI pre-audit model three is taken as the AI pre-audit data confidence;

[0038] S403, compare the AI pre-audit data with the pre-labeled result data, if the AI pre-audit data is the same as the pre-labeled result data, the AI pre-audit data is judged as the correct pre-labeled result data; if the AI pre-audit data is not the same as the pre-labeled result data, the AI pre-audit data is judged as the incorrect pre-labeled result data.

[0039] Based on the above technical scheme: by flowing the pre-annotation result data to multiple AI pre-audit models respectively, the output AI pre-audit data is not dependent on the result of one AI pre-audit model, but is obtained by comprehensively using multiple AI pre-audit models, and the AI pre-audit data has high confidence and high accuracy. In addition, the AI pre-audit data obtained by comprehensively using multiple AI pre-audit models will not cause batch errors in the pre-audit result due to the deviation of a single AI pre-audit model.

[0040] Further scheme: in step S402,

[0041] If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both less than the second threshold, the pre-annotation result data is flowed to manual audit;

[0042] If the AI pre-audit data three is different from the AI pre-audit data one and the AI pre-audit data two, the pre-annotation result data is flowed to manual audit;

[0043] If the AI pre-audit data confidence three is less than the second threshold, the pre-annotation result data is flowed to manual audit;

[0044] After manual audit, if the pre-annotation result data is judged to be incorrect, it is returned to manual annotation in step S300; if the pre-annotation result data is judged to be correct, it is passed.

[0045] Based on the above technical scheme: flowing the pre-annotation result data to manual audit for further judgment can reduce the error rate of the data passed.

[0046] Further scheme: AI pre-audit data confidence one = AI pre-audit model one original confidence * AI pre-audit model one accuracy; AI pre-audit data confidence two = AI pre-audit model two original confidence * AI pre-audit model two accuracy; AI pre-audit data confidence three = AI pre-audit model three original confidence * AI pre-audit model three accuracy.

[0047] Based on the above technical scheme: when multiple AI pre-audit models are comprehensively used to obtain AI pre-audit data, using the AI pre-audit model accuracy as a weighting coefficient can increase the weight of the AI pre-audit model with high accuracy, so that the AI pre-audit data obtained by comprehensive use is more accurate.

[0048] Further scheme: the AI pre-audit model includes AI pre-audit model one and AI pre-audit model two; the AI pre-audit step includes:

[0049] S421. The pre-annotation result data is transferred to AI pre-audit model 1 and AI pre-audit model 2 respectively. The AI ​​pre-audit model 1 obtains a plurality of pre-audit data and a plurality of pre-audit data confidence levels. The AI ​​pre-annotation model 2 obtains a plurality of pre-audit data and a plurality of pre-audit data confidence levels.

[0050] S422. Calculate the average of the pre-audit data confidence level obtained by AI pre-audit model 1 and the pre-audit data confidence level obtained by AI pre-audit model 2 for each pre-audit data, and use the pre-audit data with the highest pre-audit data confidence level as the AI ​​pre-audit data. The averaged pre-audit data confidence level is used as the AI ​​pre-audit data confidence level.

[0051] S423. Compare the AI ​​pre-review data with the pre-labeling result data. If the AI ​​pre-review data and the pre-labeling result data are the same, then determine that the AI ​​pre-review data is pre-labeling result data that has been reviewed correctly; if the AI ​​pre-review data and the pre-labeling result data are different, then determine that the AI ​​pre-review data is pre-labeling result data that has been reviewed incorrectly.

[0052] Based on the above technical solution: using AI instead of manual review can improve the efficiency of the review; and the AI ​​pre-review data obtained by integrating the two AI pre-review models has high confidence and accuracy compared to the AI ​​pre-review data obtained by a single AI pre-review model.

[0053] The beneficial effects of the present invention are:

[0054] 1. The present invention provides a pipeline data labeling method, which decomposes the overall labeling task in the labeling scheme into several labeling subtasks, and performs pre-labeling, pre-review, return to manual labeling if the review fails, and review and other operations on the data to be labeled in each labeling subtask in a pipeline labeling manner, and finally combines the reviewed data in all labeling subtasks to form the final labeling result and output it; decomposing the overall labeling task into several labeling subtasks can significantly reduce the complexity of the overall labeling task, convert it into simple labeling subtasks, facilitate learning and recognition by humans and AI, and improve the efficiency and quality of data labeling.

[0055] 2. This invention can greatly improve the efficiency of data labeling by using AI to replace manual pre-labeling and pre-review.

[0056] 3. The present invention pre-labels and pre-reviews the data to be labeled in each labeling subtask by combining manual and AI methods, which can not only improve the efficiency of data labeling, but also improve the accuracy of data labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0058] Figure 1 is a flowchart of a pipeline data labeling method of the present application.

[0059] Figure 2 is a flowchart of pre-labeling and pre-auditing of each labeling subtask of the present application.

[0060] Figure 3 is a flowchart of AI pre-labeling of the present application.

[0061] Figure 4 is a flowchart of AI pre-auditing step of the present application.

[0062] Figure 5 is a schematic diagram of building an AI pre-labeling model of the present application.

[0063] Figure 6 is a schematic diagram of building an AI pre-auditing model of the present application. DETAILED DESCRIPTION

[0064] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings of the embodiments of the present application. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. Embodiment one:

[0065] Referring to Figure 1 and Figure 2 , the present embodiment provides a pipeline data labeling method, comprising:

[0066] S100, obtaining data to be labeled and determining a labeling scheme.

[0067] Among them, obtaining data to be labeled includes uploading original unstructured data to the platform, managing the data by version on the platform, and cleaning invalid data to obtain valid data to be labeled. For example, a user uploads a face recognition data set to the platform, the platform will search and classify each data, judge the resolution, picture information, etc., and remove the obviously invalid data (such as resolution below threshold, repetition, pure color without information, etc.) to form a valid data set to be labeled.

[0068] The determining the labeling scheme comprises: determining which types of labeling (bounding box, segmentation, polyline, etc.) need to be performed on the data to be labeled, which categories of labeling objects (people, vehicles, ships, etc.) are available, and which attributes (color, age, etc.) of the labeling object categories are available.

[0069] S200, split the overall labeling task in the labeling scheme into a plurality of labeling sub-tasks.

[0070] The overall labeling task in the labeling scheme can be split into labeling sub-task A and labeling sub-task B according to the labeling type. If one-time splitting cannot achieve the finest granularity required by the pipeline, the labeling sub-task B obtained by one-time splitting can be further split according to the object and attribute to obtain labeling sub-task B1 and labeling sub-task B2. Labeling sub-task B1 and labeling sub-task B2 can also be further split to ensure that the final labeling sub-task is a simple pipeline project. The data to be labeled is copied into the pipeline of each labeling sub-task and labeled.

[0071] S300, for each labeling sub-task, according to a preset labeling mode, select a preset labeling mode; if the labeling mode is full labeling, all data to be labeled in the labeling sub-task is transferred to manual labeling;

[0072] If the labeling mode is extractive labeling, all data to be labeled in the labeling sub-task is transferred to the AI pre-labeling step to obtain AI pre-labeled data and AI pre-labeled data confidence. The AI pre-labeled data confidence is used to determine whether each AI pre-labeled data meets the condition of directly entering the AI pre-audit step. If yes, it directly enters the AI pre-audit step. If not, it is transferred to manual labeling;

[0073] The manually labeled data obtained after manual labeling is transferred to the AI pre-audit step.

[0074] The preset labeling mode includes full labeling mode and extractive labeling mode.

[0075] S400, AI pre-audit is performed on the pre-labeled result data transferred to the AI pre-audit step to obtain AI pre-audit data and AI pre-audit data confidence; the pre-labeled result data includes manually labeled data and AI pre-labeled data that is determined to meet the condition of directly entering the AI pre-audit step; the AI pre-audit data includes pre-labeled result data that is audited as incorrect and pre-labeled result data that is audited as correct; the pre-labeled result data that is audited as incorrect is transferred to manual audit;

[0076] For pre-annotated result data that have been reviewed as correct, the preset review mode will be selected according to the preset review mode; if the review mode is full review mode, all pre-annotated result data that have been reviewed as correct will be transferred to manual review; if the review mode is extraction review, the pre-annotated result data that have been reviewed as correct will be sorted from low to high according to the confidence level of the AI ​​pre-review data, and a certain proportion of the pre-annotated result data that have been reviewed as correct with a higher ranking will be extracted and transferred to manual review, and the pre-annotated result data that have been reviewed as correct with a lower ranking will be directly passed the review;

[0077] If the pre-labeled result data is judged to be incorrect after manual review, it will be returned to the manual labeling in step S300, and if the pre-labeled result data is judged to be correct, it will pass the review;

[0078] S500: Merge the reviewed data in all annotation subtasks to form the final annotation result and output it.

[0079] The merging of the data that have passed the review in all the labeling subtasks includes: re-merging the data that have passed the review in all the labeling subtasks according to the labeling plan and the splitting plan.

[0080] Based on the above scheme, in the AI ​​pre-labeling step, an AI pre-labeling model is used to pre-label all the data to be labeled in the labeling subtask.

[0081] See Figure 3 Based on any of the above solutions, in step S300, the AI ​​pre-labeling model includes AI pre-labeling model 1, AI pre-labeling model 2, and AI pre-labeling model 3; the AI ​​pre-labeling step includes:

[0082] Label all the unlabeled data in the subtask ( Figure 3 The data for the labeling subtasks are transferred to the AI ​​pre-labeling model 1 and the AI ​​pre-labeling model 2 respectively, and the AI ​​pre-labeling data 1 ( Figure 3 Model-result) and AI pre-labeled data confidence-( Figure 3 The confidence of model 1 is obtained by AI pre-labeling model 2 to obtain AI pre-labeled data 2 ( Figure 3 Model 2 results) and AI pre-labeled data confidence 2 ( Figure 3 (Medium Model 2 Confidence).

[0083] Among them, AI pre-labeled data confidence level 1 = original confidence level of AI pre-labeled model 1 * accuracy rate of AI pre-labeled model 1; AI pre-labeled data confidence level 2 = original confidence level of AI pre-labeled model 2 * accuracy rate of AI pre-labeled model 2; AI pre-labeled data confidence level 3 = original confidence level of AI pre-labeled model 3 * accuracy rate of AI pre-labeled model 3.

[0084] Wherein, the AI pre-annotation model one accuracy refers to the accuracy of the AI pre-annotation model one after being trained and tested; the AI pre-annotation model two accuracy refers to the accuracy of the AI pre-annotation model two after being trained and tested; the AI pre-annotation model three accuracy refers to the accuracy of the AI pre-annotation model three after being trained and tested.

[0085] Figure 3 The AI pre-annotation data confidence one weighting refers to calculating the AI pre-annotation data confidence one according to the weighting formula "AI pre-annotation data confidence one = AI pre-annotation model one original confidence * AI pre-annotation model one accuracy", and the same method is used for AI pre-annotation data confidence two weighting and AI pre-annotation data confidence three weighting, which will not be repeated.

[0086] In step S300, whether each piece of AI pre-annotation data meets the condition of directly entering the AI pre-audit step is determined according to the AI pre-annotation data confidence, including: comparing the AI pre-annotation data confidence one and the AI pre-annotation data confidence two with the first threshold value respectively; for example, the first threshold value can be set to 0.7;

[0087] If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both less than the first threshold value, the piece of data in the labeling sub-task is transferred to manual labeling;

[0088] If the AI pre-annotation data confidence one is greater than or equal to the first threshold value and the AI pre-annotation data confidence two is less than the first threshold value, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step;

[0089] If the AI pre-annotation data confidence two is greater than or equal to the first threshold value and the AI pre-annotation data confidence one is less than the first threshold value, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-audit step;

[0090] If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both greater than or equal to the first threshold value, whether the AI pre-annotation data one and the AI pre-annotation data two are the same is determined, if the same, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step; if not the same, the AI pre-annotation model three is introduced, and the AI pre-annotation data three (model three result) and the AI pre-annotation data confidence three (model three confidence) are obtained through the AI pre-annotation model three Figure 3 Figure 3 ​The AI pre-annotation data confidence three is compared with the first threshold value, if the AI pre-annotation data confidence three is greater than or equal to the first threshold value, it is judged whether the AI pre-annotation data three is the same as the AI pre-annotation data one or the AI pre-annotation data two, if the AI pre-annotation data three is the same as the AI pre-annotation data one, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step; if the AI pre-annotation data three is the same as the AI pre-annotation data two, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-audit step; if the AI pre-annotation data three is not the same as the AI pre-annotation data one and the AI pre-annotation data two, the piece of to-be-annotated data in the annotation subtask is transferred to manual annotation; if the AI pre-annotation data confidence three is less than the first threshold value, the piece of to-be-annotated data in the annotation subtask is transferred to manual annotation;

[0091] The manual annotation data obtained after manual annotation is transferred to the AI pre-audit step.

[0092] In step S300, the AI pre-annotation model can also be set to one, after all to-be-annotated data in the annotation subtask is transferred to the AI pre-annotation model, AI pre-annotation data and AI pre-annotation data confidence are obtained, the AI pre-annotation data confidence is directly compared with the first threshold value, if the AI pre-annotation data confidence is greater than or equal to the first threshold value, the piece of AI pre-annotation data is directly taken as the pre-annotation result data and is transferred to the AI pre-audit step; otherwise, the piece of to-be-annotated data is transferred to manual annotation.

[0093] On the basis of any of the above solutions, the AI pre-annotation model one, the AI pre-annotation model two and the AI pre-annotation model three used in the embodiment are different in AI pre-annotation algorithm, such as resnet, yolo, Googlenet, etc., because different algorithms have different emphases, the results obtained are also different, therefore, the AI pre-annotation data obtained by the AI pre-annotation model of multiple different algorithms is high in accuracy, the above three AI pre-annotation models can use existing AI pre-annotation models or can be built according to requirements, for reference Figure 5 The embodiment provides a building method of an AI pre-annotation model, comprising:

[0094] S601, obtaining pre-annotation model training set, pre-annotation model test set and pre-annotation model validation set data required by an initial AI pre-annotation model;

[0095] S602, selecting an AI pre-annotation algorithm and building the AI pre-annotation model according to an applied pre-annotation scene, such as resnet, yolo, Googlenet, etc.

[0096] S603: Training an initial AI pre-labeling model based on the selected pre-labeling model training set and the AI ​​pre-labeling algorithm;

[0097] S604: Testing the accuracy of the initial AI pre-labeling model based on the selected pre-labeling model test set, and adjusting the AI ​​pre-labeling algorithm hyperparameters according to the accuracy of the initial AI pre-labeling model to continuously improve the accuracy of the initial AI pre-labeling model;

[0098] S605. Verify the accuracy of the initial AI pre-labeling model based on the selected pre-labeling model validation set; determine whether the accuracy of the initial AI pre-labeling model meets the preset standard. If so, use the initial AI pre-labeling model as the AI ​​pre-labeling model on the pipeline; if not, adjust the pre-labeling model training set data and the AI ​​pre-labeling algorithm to retrain the initial AI pre-labeling model.

[0099] The method for building the AI ​​pre-labeling model provided in this embodiment belongs to the existing technology.

[0100] Based on any of the above solutions, in the AI ​​pre-review step, an AI pre-review model is used to perform AI pre-review on the pre-annotated result data transferred to the AI ​​pre-review step.

[0101] See Figure 4 Based on any of the above solutions, the AI ​​pre-audit model includes AI pre-audit model 1, AI pre-audit model 2, and AI pre-audit model 3; in step S400, the AI ​​pre-audit step includes:

[0102] S401, the pre-labeled result data is transferred to AI pre-audit model 1 and AI pre-audit model 2 respectively, and AI pre-audit data 1 is obtained through AI pre-audit model 1 ( Figure 4 Model-result) and AI pre-audit data confidence-level ( Figure 4 Model 1 confidence), obtain AI pre-audit data 2 through AI pre-audit model 2 ( Figure 4 Model 1 result) and AI pre-audit data confidence 2 ( Figure 4 Model 2 confidence).

[0103] Among them, AI pre-audit data confidence level 1 = original confidence level of AI pre-audit model 1 * accuracy rate of AI pre-audit model 1; AI pre-audit data confidence level 2 = original confidence level of AI pre-audit model 2 * accuracy rate of AI pre-audit model 2; AI pre-audit data confidence level 3 = original confidence level of AI pre-audit model 3 * accuracy rate of AI pre-audit model 3.

[0104] The AI pre-audit model one accuracy refers to the accuracy of the AI pre-audit model one after being trained and tested; the AI pre-audit model two accuracy refers to the accuracy of the AI pre-audit model two after being trained and tested; and the AI pre-audit model three accuracy refers to the accuracy of the AI pre-audit model three after being trained and tested.

[0105] Figure 4 The AI pre-audit data confidence one weighting refers to calculating the AI pre-audit data confidence one according to the weighting formula “AI pre-audit data confidence one = AI pre-audit model one original confidence * AI pre-audit model one accuracy”. The same method is used for AI pre-audit data confidence two weighting and AI pre-audit data confidence three weighting, which will not be repeated here.

[0106] S402, compare the AI pre-audit data confidence one and the AI pre-audit data confidence two with the second threshold value respectively; for example, the second threshold value can be set to 0.7;

[0107] If the AI pre-audit data confidence one is greater than or equal to the second threshold value and the AI pre-audit data confidence two is less than the second threshold value, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence one is taken as the AI pre-audit data confidence;

[0108] If the AI pre-audit data confidence two is greater than or equal to the second threshold value and the AI pre-audit data confidence one is less than the second threshold value, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence two is taken as the AI pre-audit data confidence;

[0109] If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both greater than or equal to the second threshold value, it is judged whether the AI pre-audit data one and the AI pre-audit data two are the same. If they are the same, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence which is higher among the AI pre-audit model one and the AI pre-audit model two is taken as the AI pre-audit data confidence. If they are not the same, the AI pre-audit model three is introduced, and the AI pre-audit data three (model three result) and the AI pre-audit data confidence three are obtained through the AI pre-audit model three. Figure 4 Figure 4 ​The AI pre-audit data confidence three is compared with the second threshold value, if the AI pre-audit data confidence three is greater than or equal to the second threshold value, whether the AI pre-audit data three is same as the AI pre-audit data one or the AI pre-audit data two is judged, if the AI pre-audit data three is same as the AI pre-audit data one, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence of the AI pre-audit model one and the AI pre-audit model three with high value is taken as the AI pre-audit data confidence; if the AI pre-audit data three is same as the AI pre-audit data two, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence of the AI pre-audit model two and the AI pre-audit model three with high value is taken as the AI pre-audit data confidence.

[0110] In step S402,

[0111] If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both less than the second threshold value, the pre-annotation result data is transferred to artificial audit.

[0112] If the AI pre-audit data three is not same as the AI pre-audit data one and the AI pre-audit data two, the pre-annotation result data is transferred to artificial audit.

[0113] If the AI pre-audit data confidence three is less than the second threshold value, the pre-annotation result data is transferred to artificial audit.

[0114] After artificial audit, if the pre-annotation result data is judged as error, the pre-annotation result data is returned to artificial annotation in step S300, and if the pre-annotation result data is judged as correct, the pre-annotation result data is passed.

[0115] S403, the AI pre-audit data is compared with the pre-annotation result data, if the AI pre-audit data is same as the pre-annotation result data, the AI pre-audit data is judged as correct pre-annotation result data, if the AI pre-audit data is not same as the pre-annotation result data, the AI pre-audit data is judged as error pre-annotation result data.

[0116] The AI pre-audit model is essentially an AI pre-annotation model.

[0117] On the basis of any of the above schemes, the AI pre-audit model one, the AI pre-audit model two and the AI pre-audit model three used in the embodiment are different in algorithm, such as resnet, yolo, Googlenet and the like, because different algorithms have different emphases, the results obtained are also different, therefore, the AI pre-audit data obtained by comprehensively using AI pre-audit models of multiple different algorithms has high accuracy, the above three AI pre-annotation models can use existing AI pre-annotation models, or can be built according to requirements, please refer toFigure 6 The embodiment provides a method for building an AI pre-audit model.

[0118] S601, obtaining pre-audit model training set, pre-audit model test set and pre-audit model verification set data required by an initial AI pre-audit model;

[0119] S602, selecting an AI pre-audit algorithm and building the AI pre-audit model according to the applied pre-audit scene, such as resnet, yolo, Googlenet, etc.

[0120] S603, training the initial AI pre-audit model based on the selected pre-audit model training set and the AI pre-audit algorithm;

[0121] S604, testing the accuracy of the initial AI pre-audit model based on the selected pre-audit model test set, and adjusting the AI pre-audit algorithm hyperparameters according to the accuracy of the initial AI pre-audit model to continuously improve the accuracy of the initial AI pre-audit model.

[0122] S605, verifying the accuracy of the initial AI pre-audit model based on the selected pre-audit model verification set; determining whether the accuracy of the initial AI pre-audit model meets a preset standard, if yes, taking the initial AI pre-audit model as an AI pre-audit model on a flow line; if not, adjusting the pre-audit model training set data and the AI pre-audit algorithm, and retraining the initial AI pre-audit model.

[0123] The method for building the AI pre-annotation model provided in the embodiment belongs to the prior art. Embodiment two

[0124] The embodiment is based on embodiment one, and is different from embodiment one in that the AI pre-annotation model includes an AI pre-annotation model one and an AI pre-annotation model two; and the AI pre-annotation step includes:

[0125] All to-be-annotated data in the annotation subtask are respectively forwarded to the AI pre-annotation model one and the AI pre-annotation model two, a plurality of pre-annotation data and a plurality of pre-annotation data confidence are obtained through the AI pre-annotation model one, and a plurality of pre-annotation data and a plurality of pre-annotation data confidence are obtained through the AI pre-annotation model two.

[0126] The pre-annotation data confidence obtained through the AI pre-annotation model one and the pre-annotation data confidence obtained through the AI pre-annotation model two are averaged for each pre-annotation data, the pre-annotation data with the highest pre-annotation data confidence is taken as a pre-annotation result data, and is forwarded to the AI pre-audit step. Embodiment three

[0127] The embodiment is based on the embodiment one, and different from the embodiment one is that the AI pre-auditing model comprises an AI pre-auditing model one and an AI pre-auditing model two; the AI pre-auditing step comprises:

[0128] S421, the pre-annotation result data is respectively transferred to the AI pre-auditing model one and the AI pre-auditing model two, a plurality of pre-auditing data and a plurality of pre-auditing data confidence are obtained through the AI pre-auditing model one, a plurality of pre-auditing data and a plurality of pre-auditing data confidence are obtained through the AI pre-annotation model two;

[0129] S422, the pre-auditing data confidence obtained by each pre-auditing data in the AI pre-auditing model one and the pre-auditing data confidence obtained by the AI pre-auditing model two are averaged, and the pre-auditing data with the highest pre-auditing data confidence is taken as the AI pre-auditing data, and the pre-auditing data confidence after averaging is taken as the AI pre-auditing data confidence;

[0130] S423, the AI pre-auditing data is compared with the pre-annotation result data, if the AI pre-auditing data is same as the pre-annotation result data, the AI pre-auditing data is judged as the pre-annotation result data which is audited to be correct, if the AI pre-auditing data is not same as the pre-annotation result data, the AI pre-auditing data is judged as the pre-annotation result data which is audited to be wrong.

[0131] The application is not limited to the optional implementation mode described above, anyone can derive other various forms of products under the inspiration of the application, but regardless of any changes in shape or structure, any technical solution falling within the scope defined by the claims of the application falls within the protection scope of the application.

Claims

1. A method for data annotation in a pipeline, characterized in that, Comprise: S100, acquire data to be labeled, and determine a labeling scheme; S200, split the overall labeling task in the labeling scheme into several labeling subtasks; S300, for each labeling subtask, according to a preset labeling mode, select to enter the preset labeling mode; if the labeling mode is full labeling, flow all the data to be labeled in the labeling subtask to artificial labeling; If the labeling mode is extraction labeling, flow all the data to be labeled in the labeling subtask to an AI pre-labeling step to obtain AI pre-labeled data and AI pre-labeled data confidence; determine whether each AI pre-labeled data meets the condition of directly entering the AI pre-audit step according to the AI pre-labeled data confidence, if yes, directly enter the AI pre-audit step; if not, flow to artificial labeling; After artificial labeling, the artificial labeled data are flowed to the AI pre-audit step; S400, AI pre-audit the pre-labeled result data flowed to the AI pre-audit step to obtain AI pre-audit data and AI pre-audit data confidence; the pre-labeled result data includes artificial labeled data and AI pre-labeled data determined to meet the condition of directly entering the AI pre-audit step; the AI pre-audit data includes pre-labeled result data audited as incorrect and pre-labeled result data audited as correct; for the pre-labeled result data audited as incorrect, flow to artificial audit; For the pre-labeled result data audited as correct, according to a preset audit mode, select to enter the preset audit mode; if the audit mode is full audit mode, flow all the pre-labeled result data audited as correct to artificial audit; if the audit mode is extraction audit, according to the AI pre-audit data confidence, sort the pre-labeled result data audited as correct from low to high, and extract a certain proportion of pre-labeled result data audited as correct at the top to flow to artificial audit, and for the pre-labeled result data audited as correct at the bottom, directly pass the audit; After artificial audit, the pre-labeled result data determined as incorrect is returned to artificial labeling in step S300, and the pre-labeled result data determined as correct passes the audit; S500, combine the data passing the audit in all labeling subtasks to form the final labeling result and output.

2. The data labeling method of claim 1, wherein in the AI pre-labeling step, an AI pre-labeling model is used to pre-label all the data to be labeled in the labeling subtask. The AI pre-labeling model comprises an AI pre-labeling model one, an AI pre-labeling model two and an AI pre-labeling model three; the AI pre-labeling step comprises:

3. The method of claim 2, wherein, Flow all the data to be labeled in the labeling subtask to the AI pre-labeling model one and the AI pre-labeling model two respectively, obtain AI pre-labeled data one and AI pre-labeled data confidence one through the AI pre-labeling model one, and obtain AI pre-labeled data two and AI pre-labeled data confidence two through the AI pre-labeling model two; ​ In step S300, judging whether each AI pre-annotation data meets the condition of directly entering the AI pre-audit step according to the AI pre-annotation data confidence includes: comparing the AI pre-annotation data confidence one and the AI pre-annotation data confidence two with the first threshold respectively; If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both less than the first threshold, the piece of to-be-annotated data in the annotation sub-task is transferred to manual annotation; If the AI pre-annotation data confidence one is greater than or equal to the first threshold and the AI pre-annotation data confidence two is less than the first threshold, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step; If the AI pre-annotation data confidence two is greater than or equal to the first threshold and the AI pre-annotation data confidence one is less than the first threshold, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-audit step; If the AI pre-annotation data confidence one and the AI pre-annotation data confidence two are both greater than or equal to the first threshold, it is judged whether the AI pre-annotation data one and the AI pre-annotation data two are the same, if the same, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step, if not the same, the AI pre-annotation model three is introduced, the AI pre-annotation data three and the AI pre-annotation data confidence three are obtained through the AI pre-annotation model three, the AI pre-annotation data confidence three is compared with the first threshold, if the AI pre-annotation data confidence three is greater than or equal to the first threshold, it is judged whether the AI pre-annotation data three is the same as the AI pre-annotation data one or the AI pre-annotation data two, if the AI pre-annotation data three is the same as the AI pre-annotation data one, the AI pre-annotation data one is taken as the pre-annotation result data and is transferred to the AI pre-audit step, if the AI pre-annotation data three is the same as the AI pre-annotation data two, the AI pre-annotation data two is taken as the pre-annotation result data and is transferred to the AI pre-audit step, if the AI pre-annotation data three is not the same as the AI pre-annotation data one and the AI pre-annotation data two, the piece of to-be-annotated data in the annotation sub-task is transferred to manual annotation, if the AI pre-annotation data confidence three is less than the first threshold, the piece of to-be-annotated data in the annotation sub-task is transferred to manual annotation; The manual annotation data obtained after manual annotation is transferred to the AI pre-audit step.

4. The method of claim 3, wherein, The AI pre-annotation data confidence one = AI pre-annotation model one original confidence * AI pre-annotation model one accuracy; the AI pre-annotation data confidence two = AI pre-annotation model two original confidence * AI pre-annotation model two accuracy; the AI pre-annotation data confidence three = AI pre-annotation model three original confidence * AI pre-annotation model three accuracy.

5. The method of claim 2, wherein, The AI pre-annotation model includes the AI pre-annotation model one and the AI pre-annotation model two; the AI pre-annotation step includes: The AI pre-labeling model one and the AI pre-labeling model two are respectively used to pre-label all the to-be-labeled data in the subtask, the AI pre-labeling model one is used to obtain a plurality of pre-labeled data and a plurality of pre-labeled data confidence, and the AI pre-labeling model two is used to obtain a plurality of pre-labeled data and a plurality of pre-labeled data confidence; The pre-labeled data confidence obtained by the AI pre-labeling model one and the pre-labeled data confidence obtained by the AI pre-labeling model two are averaged, the pre-labeled data with the highest pre-labeled data confidence is taken as the pre-labeling result data, and the pre-labeling result data is transferred to the AI pre-audit step.

6. The pipeline data labeling method according to claim 1, characterized in that, In the AI pre-audit step, an AI pre-audit model is used to pre-audit the pre-labeling result data transferred to the AI pre-audit step.

7. The method of claim 6, wherein, The AI pre-audit model comprises an AI pre-audit model one, an AI pre-audit model two and an AI pre-audit model three; and the AI pre-audit step comprises: S401. The pre-labeling result data is respectively transferred to the AI pre-audit model one and the AI pre-audit model two, the AI pre-audit data one and the AI pre-audit data confidence one are obtained by using the AI pre-audit model one, and the AI pre-audit data two and the AI pre-audit data confidence two are obtained by using the AI pre-audit model two; S402. The AI pre-audit data confidence one and the AI pre-audit data confidence two are respectively compared with a second threshold value; If the AI pre-audit data confidence one is greater than or equal to the second threshold value and the AI pre-audit data confidence two is less than the second threshold value, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence one is taken as the AI pre-audit data confidence; If the AI pre-audit data confidence two is greater than or equal to the second threshold value and the AI pre-audit data confidence one is less than the second threshold value, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence two is taken as the AI pre-audit data confidence; If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both greater than or equal to the second threshold value, it is determined whether the AI pre-audit data one and the AI pre-audit data two are the same, if they are the same, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence which is higher in the AI pre-audit model one and the AI pre-audit model two is taken as the AI pre-audit data confidence; if they are not the same, the AI pre-audit model three is introduced, the AI pre-audit data three and the AI pre-audit data confidence three are obtained through the AI pre-audit model three, the AI pre-audit data confidence three is compared with the second threshold value, if the AI pre-audit data confidence three is greater than or equal to the second threshold value, it is determined whether the AI pre-audit data three is the same as the AI pre-audit data one or the AI pre-audit data two, if the AI pre-audit data three is the same as the AI pre-audit data one, the AI pre-audit data one is taken as the AI pre-audit data, and the AI pre-audit data confidence which is higher in the AI pre-audit model one and the AI pre-audit model three is taken as the AI pre-audit data confidence; if the AI pre-audit data three is the same as the AI pre-audit data two, the AI pre-audit data two is taken as the AI pre-audit data, and the AI pre-audit data confidence which is higher in the AI pre-audit model two and the AI pre-audit model three is taken as the AI pre-audit data confidence; S403, the AI pre-audit data is compared with the pre-labeled result data, if the AI pre-audit data is the same as the pre-labeled result data, the AI pre-audit data is determined as the pre-labeled result data which is audited as correct; if the AI pre-audit data is not the same as the pre-labeled result data, the AI pre-audit data is determined as the pre-labeled result data which is audited as incorrect.

8. The method of claim 7, wherein, In step S402, If the AI pre-audit data confidence one and the AI pre-audit data confidence two are both less than the second threshold value, the pre-labeled result data is transferred to artificial audit; If the AI pre-audit data three is not the same as the AI pre-audit data one and the AI pre-audit data two, the pre-labeled result data is transferred to artificial audit; If the AI pre-audit data confidence three is less than the second threshold value, the pre-labeled result data is transferred to artificial audit; After artificial audit, the pre-labeled result data which is determined as incorrect is returned to artificial labeling in step S300, and the pre-labeled result data which is determined as correct is passed through audit.

9. The method of claim 7, wherein, The AI pre-audit data confidence one is AI pre-audit model one original confidence*AI pre-audit model one accuracy rate; the AI pre-audit data confidence two is AI pre-audit model two original confidence*AI pre-audit model two accuracy rate; and the AI pre-audit data confidence three is AI pre-audit model two original confidence*AI pre-audit model three accuracy rate.

10. The method of claim 6, wherein, The AI pre-audit model includes the AI pre-audit model one and the AI pre-audit model two; and the AI pre-audit step includes: S421. The pre-annotation result data is transferred to AI pre-audit model 1 and AI pre-audit model 2 respectively. The AI ​​pre-audit model 1 obtains a plurality of pre-audit data and a plurality of pre-audit data confidence levels. The AI ​​pre-annotation model 2 obtains a plurality of pre-audit data and a plurality of pre-audit data confidence levels. S422. Calculate the average of the pre-audit data confidence level obtained by AI pre-audit model 1 and the pre-audit data confidence level obtained by AI pre-audit model 2 for each pre-audit data, and use the pre-audit data with the highest pre-audit data confidence level as the AI ​​pre-audit data. The averaged pre-audit data confidence level is used as the AI ​​pre-audit data confidence level. S423. Compare the AI ​​pre-review data with the pre-labeling result data. If the AI ​​pre-review data and the pre-labeling result data are the same, then determine that the AI ​​pre-review data is pre-labeling result data that has been reviewed correctly; if the AI ​​pre-review data and the pre-labeling result data are different, then determine that the AI ​​pre-review data is pre-labeling result data that has been reviewed incorrectly.

Images