Label data quality inspection method, server and storage medium

Abstract

The embodiment of the invention provides an annotation data quality inspection method, a server and a storage medium, and belongs to the field of data annotation. The method comprises the steps of obtaining to-be-detected annotation data containing a thunder file; determining a consistency rate between a standard labeling result of second to-be-labeled data in the thunder file and a to-be-detected labeling result of the same second to-be-labeled data; and according to the consistency rate, determining a first annotation correct rate of the data annotation task, and according to the first annotation correct rate, determining a quality inspection result of the to-be-inspected annotated data. According to the embodiment of the invention, the quality inspection efficiency and accuracy of the annotation data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data annotation, and in particular to a quality inspection method, a server and a storage medium for annotated data. Background Art

[0002] Data annotation is crucial in the field of artificial intelligence because it provides machine learning models with precisely labeled training data, enabling them to recognize patterns, understand inputs, and learn effectively, thereby achieving high-precision and high-performance artificial intelligence functions in various applications such as image recognition, natural language processing, and autonomous vehicles. Unlabeled or inaccurately labeled data will lead to reduced model effectiveness, so data annotation is an indispensable part of building a powerful artificial intelligence system.

[0003] In order to make the annotation results of training data more accurate, professional data annotators in various fields are usually required to annotate the training data, and combine relevant data inspection methods to conduct quality inspection on the annotation results. At present, the quality inspection of large-scale data annotation is mainly carried out manually by quality inspectors. For example, the quality of large-scale data annotation is guaranteed by designing multiple rounds of quality inspection by different personnel during the annotation process. However, multiple quality inspectors are required, and the entire quality inspection process is cumbersome. The quality inspection efficiency of the labeled data is low, and the accuracy of the quality inspection of the labeled data cannot be effectively guaranteed. Therefore, how to improve the accuracy and efficiency of the quality inspection of labeled data is a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a quality inspection method for labeled data, a server, and a storage medium, aiming to improve the accuracy and efficiency of quality inspection of labeled data.

[0005] In a first aspect, an embodiment of the present invention provides a method for quality inspection of labeled data, comprising: Acquire the labeled data to be inspected containing a Lei file, wherein the labeled data to be inspected is generated by a data annotator after completing a data labeling task containing the Lei file and the first data to be labeled, the Lei file includes the second data to be labeled and the standard labeling result of the second data to be labeled, and the labeled data to be inspected includes the labeling result to be inspected of the second data to be labeled; Determine a consistency rate between the standard annotation result of the second data to be annotated in the thunder file and the to-be-checked annotation result of the same second data to be annotated; A first annotation accuracy rate of the data annotation task is determined according to the consistency rate, and a quality inspection result of the to-be-inspected annotated data is determined according to the first annotation accuracy rate.

[0006] In a second aspect, an embodiment of the present invention further provides a server, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the quality inspection method for labeled data as described in the first aspect is implemented.

[0007] In a third aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the quality inspection method for labeled data as described in the first aspect.

[0008] The embodiment of the present invention provides a quality inspection method, server and storage medium for labeled data. The labeled data to be inspected in the embodiment of the present invention contains a thunder file, so that the consistency rate between the standard labeling result of the data to be labeled in the thunder file and the labeling result to be inspected of the same data to be labeled can be conveniently and automatically determined, and then the labeling accuracy of the data labeling task is determined according to the consistency rate. Finally, the quality inspection result of the labeled data to be inspected is determined according to the labeling accuracy of the data labeling task. The whole process does not require the participation of quality inspectors, effectively improves the quality inspection efficiency of the labeled data, and effectively avoids the misjudgment caused by the subjective reasons of the quality inspectors in manual quality inspection, thereby improving the accuracy of the labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 It is a flowchart of a quality inspection method for labeled data provided by an embodiment of the present invention; Figure 2 is a schematic diagram of an audio segmentation result in an embodiment of the present invention; Figure 3 is another schematic diagram of an audio segmentation result in an embodiment of the present invention; Figure 4 It is a flowchart of another quality inspection method for labeled data provided by an embodiment of the present invention; Figure 5 It is a flowchart of another quality inspection method for labeled data provided by an embodiment of the present invention; Figure 6 is a schematic diagram of the labeling and quality inspection process in an embodiment of the present invention; Figure 7 is another schematic diagram of the labeling and quality inspection process in an embodiment of the present invention; Figure 8 The present invention is a schematic structural block diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0013] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0014] At present, the quality inspection of large-scale data annotation is mainly carried out manually by quality inspectors. For example, the quality of large-scale data annotation is ensured by designing multiple rounds of quality inspection by different personnel during the annotation process. However, it requires multiple quality inspectors, and the entire quality inspection process is cumbersome. The quality inspection efficiency of the labeled data is low, and the accuracy of the quality inspection of the labeled data cannot be effectively guaranteed. Therefore, how to improve the accuracy and efficiency of the quality inspection of the labeled data is an urgent problem to be solved.

[0015] To solve the above problems, the embodiments of the present invention provide a quality inspection method, server and storage medium for labeled data. The labeled data to be inspected in the embodiments of the present invention contains a thunder file, so that the consistency rate between the standard annotation results of the data to be labeled in the thunder file and the inspection annotation results of the same data to be labeled can be conveniently and automatically determined, and then the annotation accuracy of the data annotation task is determined according to the consistency rate, and finally the quality inspection result of the labeled data to be inspected is determined according to the annotation accuracy of the data annotation task. The whole process does not require the participation of quality inspectors, which effectively improves the quality inspection efficiency of the labeled data, and effectively avoids the misjudgment caused by subjective reasons of quality inspectors in manual quality inspection, thereby improving the accuracy of the labeled data.

[0016] Among them, the quality inspection method of labeled data can be applied to a server, which can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0017] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0018] Please refer to Figure 1 , Figure 1 A flow chart of a method for quality inspection of annotated data provided by an embodiment of the present invention. The method for quality inspection of annotated data can be applied to a computer device. The computer device can include a server or a terminal device.

[0019] like Figure 1 As shown, the quality inspection method for annotated data includes steps S101 to S104.

[0020] Step S101, obtaining the labeled data to be inspected containing the thunder file.

[0021] In this embodiment, the Lei file refers to a file including the data to be labeled and the standard labeling results of the data to be labeled (that is, the correct labeling results of the data to be labeled). In this embodiment, the Lei file includes the second data to be labeled and the standard labeling results of the second data to be labeled. The labeled data to be checked is generated by the data annotator after completing the data labeling task containing the Lei file and the first data to be labeled, that is, the labeled data to be checked is obtained by the data annotator by labeling the first data to be labeled and the second data to be labeled contained in the data labeling task, and the labeled data to be checked includes the labeling results of the second data to be labeled.

[0022] In some embodiments, the first data to be labeled or the second data to be labeled may be unimodal data, for example, the first data to be labeled is image data or audio data, and the second data to be labeled is image data or audio data. The first data to be labeled or the second data to be labeled may also be multimodal data, for example, the first data to be labeled includes image data and audio data, and the second data to be labeled includes image data or audio data.

[0023] In some embodiments, obtaining the labeled data to be checked containing the Lei file may include: obtaining the labeled data to be checked containing the Lei file sent by the terminal device, wherein the terminal device responds to the submission instruction of the data labeling task triggered by the data labeler and sends the labeled data to be checked corresponding to the data labeling task to the server. The labeled data to be checked includes the Lei file and multiple labeled results to be checked, and the multiple labeled results to be checked include the labeled results to be checked of the second data to be labeled in the Lei file.

[0024] In some embodiments, the frequency of occurrence of the Lei file in the data labeling task during the data labeling process of the data labeler is negatively correlated with the labeling accuracy of the data labeler. That is, the higher the labeling accuracy of the data labeler, the lower the frequency of occurrence of the Lei file in the data labeling task during the data labeling process of the data labeler, and the lower the labeling accuracy of the data labeler, the higher the frequency of occurrence of the Lei file in the data labeling task during the data labeling process of the data labeler. Since the frequency of occurrence of the Lei file in the data labeler's data labeling process is negatively correlated with the labeling accuracy of the data labeler, for novices or data labelers with low labeling accuracy, the accuracy of quality inspection can be ensured by adding more Lei files to the data labeling task, and for data labelers with high labeling accuracy, the efficiency of quality inspection can be improved by adding fewer Lei files to the data labeling task, and the accuracy of quality inspection can be ensured, so as to achieve a balance between efficiency and accuracy of quality inspection.

[0025] Step S102: Determine the consistency rate between the standard annotation result of the second data to be annotated in the thunder file and the to-be-checked annotation result of the same second data to be annotated.

[0026] In this embodiment, the labeled data to be checked includes one or more Lei files. When the labeled data to be checked includes one Lei file, it is only necessary to determine the consistency rate between the standard labeling result of the second data to be labeled in this Lei file and the labeling result to be checked of the same second data to be labeled, and obtain the single labeling consistency rate of this Lei file. When the labeled data to be checked includes multiple Lei files, it is necessary to determine the consistency rate between the standard labeling result of the second data to be labeled in each Lei file and the labeling result to be checked of the same second data to be labeled, and obtain the single labeling consistency rate corresponding to each Lei file. Among them, the consistency rate between the standard labeling result and the labeling result to be checked may include the consistency rate of the classification label, the consistency rate of the text, the consistency rate of the audio segmentation, the consistency rate of the rectangular box, the consistency rate of the voice, the consistency rate of the contour, the consistency rate of the key points (for example, facial feature points or bone points) and the consistency rate of the semantic segmentation map.

[0027] In some embodiments, the standard annotation result includes a first classification label, the to-be-checked annotation result includes a second classification label, and the consistency rate between the first classification label and the second classification label is determined. : Determine the total number of dimensions of the first classification label, recorded as ; Compare the first classification label with the second classification label of the same dimension to determine the number of dimensions whose first classification labels are consistent with the second classification labels, recorded as , the number of dimensions whose first category label is consistent with the second category label Divide by the total number of dimensions to get the consistency rate of the classification label .

[0028] In some embodiments, the standard annotation result includes a first text, the to-be-checked annotation result includes a second text, and the text consistency rate between the first text and the second text is determined. This may include: determining a minimum edit distance between the first text and the second text, denoted as ; The larger value of the number of characters in the first text and the number of characters in the second text is determined as the target value, or the larger value of the number of words in the first text and the number of words in the second text is determined as the target value, recorded as ; According to the target value and the minimum edit distance , determine the text consistency rate . Wherein, determining the minimum edit distance between the first text and the second text may include: for each character or word in the first text, calculating the number of edit operations required to replace it with the character or word at the corresponding position in the second text; for each character or word in the second text, calculating the number of edit operations required to replace it with the character or word at the corresponding position in the first text; and determining the minimum value of the number of edit operations corresponding to each character or word in the first text and the number of edit operations corresponding to each character or word in the second text as the minimum edit distance between the first text and the second text. It can be understood that the editing operation includes adding characters, deleting characters, and replacing characters. The smaller the edit distance, the greater the similarity between the two texts. If the first text and the second text are exactly the same, their edit distance is 0. If one of the first text and the second text is empty, and the other text in the first text and the second text is not empty, their edit distance is the length of the other text. If both the first text and the second text are empty, their edit distance is 0.

[0029] For example, as shown in Table 1, the minimum edit distance between the first text "What are you doing" and the second text "What are you doing" is 1, and the text consistency rate is 0.8, the minimum edit distance between the first text "What are you doing" and the second text "What are you doing" is 1, and the text consistency rate is 0.8333, and the minimum edit distance between the first text "What are you doing" and the second text "What are you doing" is 1, and the text consistency rate is 0.8333.

[0030] Table 1

[0031] For another example, as shown in Table 2, the minimum edit distance between the first text "What are you doing" and the second text "What are youdoing now" is 1, and the text consistency rate is 0.8; the minimum edit distance between the first text "What is you doing" and the second text "What are you doing" is 1, and the text consistency rate is 0.75.

[0032] Table 2

[0033] In some embodiments, the standard annotation result includes a first audio segmentation result, the to-be-checked annotation result includes a second audio segmentation result, and the consistency rate between the first audio segmentation result and the second audio segmentation result is determined. It may include: determining the start and end time points of each first audio segment in the first audio segmentation result and determining the start and end time points of each second audio segment in the second audio segmentation result; determining the number of target audio segment groups according to the start and end time points of each audio segment in the first audio segmentation result and the start and end time points of each audio segment in the second audio segmentation result, the target audio segment group including a first audio segment and a second audio segment, and the first audio segment is segmented consistently with the second audio segment; dividing the number of target audio segment groups by the larger of the number of audio segments in the first audio segmentation result and the number of audio segments in the second audio segmentation result to obtain a consistency rate of audio segmentation.

[0034] In some embodiments, the consistent segmentation of the first audio segment and the second audio segment includes that the start and end time points of the first audio segment are the same or approximately the same as the start and end time points of the second audio segment. Among them, the start and end time points of the first audio segment are approximately the same as the start and end time points of the second audio segment include that the absolute value of the difference between the start time point of the first audio segment and the start time point of the second audio segment is within a preset range, and the absolute value of the difference between the end time point of the first audio segment and the end time point of the second audio segment is within a preset range. The preset range can be set based on actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset range is 0-500 milliseconds or 0-300 milliseconds, etc.

[0035] For example, Figure 2As shown in Table 3, the first audio segment includes the first audio segment 1 and the first audio segment 2, the second audio segment includes the second audio segment 1, the second audio segment 2 and the third audio segment 3, and the start and end time points of the first audio segment 1 are 0-a, the start and end time points of the first audio segment 2 are ac, the start and end time points of the second audio segment 1 are 0-a, the start and end time points of the second audio segment 2 are ab, and the start and end time points of the third audio segment 3 are bc. As shown in Table 3, the start and end time points of the first audio segment 1 are the same as the start and end time points of the second audio segment 1, the start and end time points of the first audio segment 2 are different from the start and end time points of the second audio segment 2, and the start and end time points of the first audio segment 2 are different from the start and end time points of the second audio segment 3, then the consistency rate of audio segmentation is .

[0036] Table 3

[0037] For example, Figure 3 As shown, the first audio segment includes the first audio segment 1 and the first audio segment 2, the second audio segment includes the second audio segment 1 and the second audio segment 2, the start and end time points of the first audio segment 1 are 0-d, the start and end time points of the first audio segment 2 are df, the start and end time points of the second audio segment 1 are 0-e, and the start and end time points of the second audio segment 2 are ef. As shown in Table 4, the start time point of the first audio segment 1 is the same as the start time point of the second audio segment 1 (both are 0), and the absolute value of the difference between the end time point d of the first audio segment 1 and the end time point e of the second audio segment 1 is ed<500 milliseconds, then the first audio segment 1 is consistent with the second audio segment 1, the absolute value of the difference between the start time point d of the first audio segment 2 and the start time point f of the second audio segment 2 is ed<500 milliseconds, and the end time point of the first audio segment 2 is the same as the end time point of the second audio segment 2 (both are f), then the first audio segment 2 is consistent with the second audio segment 2, therefore, the consistency rate of audio segmentation is .

[0038] Table 4

[0039] For example, as shown in Table 5, the start time point of the first audio segment 1 is the same as the start time point of the second audio segment 1 (both are 0), and the absolute value of the difference between the end time point d of the first audio segment 1 and the end time point e of the second audio segment 1 is ed>500 milliseconds, then the first audio segment 1 is inconsistent with the second audio segment 1, the absolute value of the difference between the start time point d of the first audio segment 2 and the start time point f of the second audio segment 2 is ed>500 milliseconds, and the end time point of the first audio segment 2 is the same as the end time point of the second audio segment 2 (both are f), then the first audio segment 2 is inconsistent with the second audio segment 2, therefore, the consistency rate of audio segmentation is .

[0040] Table 5

[0041] In some embodiments, the standard annotation result includes a first rectangular box, the to-be-checked annotation result includes a second rectangular box, and the consistency rate between the first rectangular box and the second rectangular box is determined. The method may include: determining the area of ​​the overlapping region between the first rectangular frame and the second rectangular frame, which is recorded as θ; and dividing the area of ​​the overlapping region between the first rectangular frame and the second rectangular frame by the larger of the area of ​​the first rectangular frame (recorded as θ1) and the area of ​​the second rectangular frame (recorded as θ2) to obtain the consistency rate of the rectangular frames. .

[0042] In some embodiments, the standard annotation result includes multiple first annotation results, and the types of the multiple first annotation results are different. The annotation result to be checked includes multiple second annotation results, and the types of the multiple second annotation results are different. Determining the consistency rate between the standard annotation result of the second data to be labeled in the Lei file and the annotation result to be checked of the same second data to be labeled includes: determining the consistency rate between the first annotation result of each type of the second data to be labeled in the Lei file and the second annotation result of the same type; performing weighted summation on the consistency rate of each type of the first annotation result and the second annotation result of the same type to obtain the consistency rate between the standard annotation result and the annotation result to be checked. This embodiment can accurately determine the consistency rate between the standard annotation result and the annotation result to be checked.

[0043] For example, the consistency rate between the standard annotation result and the to-be-tested annotation result can be expressed as , where n is the larger of the number of the first annotation results and the number of the second annotation results, is the consistency rate between the first annotation result of the i-th type and the second annotation result of the i-th type, is the weighting coefficient corresponding to the i-th type. The weighting coefficients of different types may be the same or different. i is an integer greater than or equal to 1 and less than n.

[0044] Step S103: Determine the first annotation accuracy of the data annotation task according to the consistency rate.

[0045] In this embodiment, when there is only one Lei file in the data labeling task, the consistency rate between the standard labeling result of the second data to be labeled in the Lei file and the check labeling result of the same second data to be labeled is determined as the first labeling accuracy rate of the data labeling task.

[0046] In some embodiments, when there are multiple Lei files in the data labeling task, the average consistency rate is calculated according to the consistency rate between the standard labeling result of the second data to be labeled in each Lei file and the labeling result to be checked of the same second data to be labeled and the number of Lei files in the data labeling task, and the average consistency rate is determined as the first labeling accuracy rate. For example, the consistency rate between the standard labeling result of the second data to be labeled in each Lei file and the labeling result to be checked of the same second data to be labeled is summed to obtain the total consistency rate, and the total consistency rate is divided by the number of Lei files in the data labeling task to obtain the average consistency rate, that is, the first labeling accuracy rate of the data labeling task.

[0047] Step S104: Determine the quality inspection result of the labeled data to be inspected according to the first labeling accuracy rate.

[0048] This embodiment obtains the labeled data to be inspected that includes the Lei file, so that it can conveniently and automatically determine the consistency rate between the standard labeling results of the data to be inspected in the Lei file and the labeling results of the same data to be inspected, and then determine the labeling accuracy of the data labeling task based on the consistency rate. Finally, based on the labeling accuracy of the data labeling task, determine the quality inspection result of the labeled data to be inspected. The whole process does not require the participation of quality inspectors, effectively improves the quality inspection efficiency of the labeled data, and effectively avoids misjudgment caused by subjective reasons of quality inspectors in manual quality inspection, thereby improving the accuracy of the labeled data.

[0049] In some embodiments, when the first annotation accuracy is greater than or equal to the preset accuracy threshold, the quality inspection result of the labeled data to be inspected is determined to be that the labeled data to be inspected has passed the quality inspection, and when the first annotation accuracy is less than the first annotation accuracy, the quality inspection result of the labeled data to be inspected is determined to be that the labeled data to be inspected has failed the quality inspection. The preset accuracy threshold can be set based on actual conditions, and the embodiments of the present invention do not specifically limit this.

[0050] In some embodiments, when the first annotation accuracy is less than a preset threshold, a first prompt message is pushed to the data annotator, and the first prompt message is used to prompt that the annotated data to be inspected has not passed the quality inspection; when the quality inspection result is that the first annotation accuracy is greater than or equal to the preset threshold, if the annotated data to be inspected is submitted for the first time, a second prompt message is pushed to the data annotator, and the second prompt message is used to prompt that the annotated data to be inspected has passed the quality inspection, without manual quality inspection. If the annotated data to be inspected is not submitted for the first time, the annotated data to be inspected will be pushed to the quality inspector for manual quality inspection.

[0051] In some embodiments, Figure 4 As shown, before step S101, it also includes: Step S105: Obtain the mine-laying ratio corresponding to the data labeler.

[0052] In this embodiment, when assigning a data annotation task to a data annotator or when the data annotator receives a data annotation task, the mine-laying ratio corresponding to the data annotator can be obtained. The mine-laying ratio is the ratio of the number of mine files inserted into the data annotation task to the number of first data to be annotated in the data annotation task.

[0053] In some embodiments, the mine-laying ratio is within a preset ratio range, which can be flexibly adjusted based on factors such as specific task requirements (the completion deadline of the entire data labeling task and the number of data labelers, etc.), budget, and the quality of the labeled data, and the embodiments of the present invention do not specifically limit this. For example, the preset ratio range is 10% to 30%. This embodiment makes the mine-laying ratio within the preset ratio range to take into account the overall efficiency and labeling quality of data labeling.

[0054] In some embodiments, obtaining the mine-laying ratio corresponding to the data annotator may include: obtaining the second annotation accuracy of the data annotator; and determining the mine-laying ratio corresponding to the data annotator based on the second annotation accuracy. The second annotation accuracy is determined based on the first annotation accuracy of each data annotation task completed by the data annotator. For example, the first annotation accuracy of each data annotation task completed by the data annotator is accumulated, and then divided by the number of data annotation tasks completed by the data annotator to obtain the second annotation accuracy of the data annotator. This embodiment can set a matching mine-laying ratio for the data annotator through the second annotation accuracy of the data annotator, realize personalized quality inspection of the data annotator's labeled data, and effectively improve the flexibility and accuracy of quality inspection.

[0055] In some embodiments, the mine-burying ratio is within a preset ratio range, and the mine-burying ratio of the mine file is negatively correlated with the second annotation accuracy of the data annotator. That is, the higher the second annotation accuracy of the data annotator, the lower the mine-burying ratio of the mine file, so that the mine file in the data annotation task has a lower frequency of occurrence during the data annotator's data annotation process, and the lower the data annotator's annotation accuracy, the higher the mine-burying ratio of the mine file, so that the mine file in the data annotation task has a higher frequency of occurrence during the data annotator's data annotation process. This embodiment can ensure the accuracy of quality inspection of the labeled data of the data annotator with a lower annotation accuracy, and can improve the efficiency of quality inspection of the labeled data of the data annotator with a higher annotation accuracy, so as to achieve a balance between efficiency and accuracy of quality inspection.

[0056] In some embodiments, the quality inspection method provided by the embodiment of the present invention further includes: adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the first annotation accuracy of the data annotation task; and / or adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the manual sampling accuracy of the data annotation task, the manual sampling accuracy being determined based on the sampling annotation results obtained by the quality inspector from annotating part of the first data to be annotated in the annotated data to be inspected corresponding to the data annotation task and the to-be-inspected annotation results of part of the first data to be annotated. This embodiment can improve the accuracy of quality inspection by adjusting the proportion of buried mines and / or the difficulty level of the mine file through the first annotation accuracy of the data annotation task and / or the manual sampling accuracy.

[0057] For example, the quality inspector extracts a plurality of first labeled data to be inspected from the labeled data to be inspected corresponding to the data annotation task for annotation, and obtains the sampled labeled data, which includes the sampled labeled results of the plurality of first labeled data to be inspected. The consistency rate between the sampled labeled result of each first labeled data to be inspected in the sampled labeled data and the labeled result to be inspected of the corresponding first labeled data to be inspected is determined, and the consistency rates between the sampled labeled result of each first labeled data to be inspected in the sampled labeled data and the labeled result to be inspected of the corresponding first labeled data to be inspected are accumulated to obtain the total consistency rate, and then the total consistency rate is divided by the number of the extracted first labeled data to be inspected to obtain the manual sampling accuracy rate of the data annotation task.

[0058] In some embodiments, adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the first annotation accuracy of the data annotation task may include: when the first annotation accuracy of the data annotation task is greater than or equal to the first accuracy threshold, lowering the proportion of buried mines and / or raising the difficulty level of the mine file. In this embodiment, when the first annotation accuracy is high, the proportion of buried mines is lowered, thereby reducing the frequency of occurrence of the mine file in the process of data annotation by the data annotator, thereby improving the overall efficiency, and when the first annotation accuracy is high, the difficulty level of the mine file is increased, thereby increasing the difficulty of the data annotator to annotate the data to be annotated in the mine file, thereby improving the accuracy of quality inspection.

[0059] In some embodiments, adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the first annotation accuracy of the data annotation task may include: when the first annotation accuracy of the data annotation task is less than the second accuracy threshold, increasing the proportion of buried mines and / or decreasing the difficulty level of the mine file. Among them, the first accuracy threshold is greater than the second accuracy threshold, and the first accuracy threshold and the second accuracy threshold can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. This embodiment increases the frequency of occurrence of the mine file in the process of data annotation by the data annotator in the data annotation process to improve the accuracy of quality inspection, and reduces the difficulty level of the mine file when the first annotation accuracy is low, thereby reducing the difficulty of the data annotator in annotating the data to be annotated in the mine file, so as to improve the overall efficiency.

[0060] In some embodiments, adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the manual sampling accuracy of the data labeling task may include: when the manual sampling accuracy of the data labeling task is greater than or equal to the first accuracy threshold, lowering the proportion of buried mines and / or raising the difficulty level of the mine file. This embodiment reduces the frequency of occurrence of the mine file in the process of data labeling by the data labeler by lowering the proportion of buried mines when the manual sampling accuracy is high, thereby improving the overall efficiency, and increases the difficulty level of the mine file when the manual sampling accuracy is high, thereby increasing the difficulty of the data labeler to label the data to be labeled in the mine file, so as to improve the accuracy of quality inspection.

[0061] In some embodiments, adjusting the proportion of buried mines and / or the difficulty level of the mine file according to the manual sampling accuracy of the data labeling task may include: when the manual sampling accuracy of the data labeling task is less than the second accuracy threshold, increasing the proportion of buried mines and / or decreasing the difficulty level of the mine file. In this embodiment, when the manual sampling accuracy is low, the proportion of buried mines is increased, thereby increasing the frequency of occurrence of the mine file in the process of data labeling by the data labeler to improve the accuracy of quality inspection, and when the manual sampling accuracy is low, the difficulty level of the mine file is lowered, thereby reducing the difficulty of the data labeler to label the data to be labeled in the mine file, thereby improving the overall efficiency.

[0062] In some embodiments, adjusting the mine-burying proportion and / or the difficulty level of the mine file according to the first annotation accuracy rate and the manual sampling accuracy rate of the data annotation task may include: when any one of the first annotation accuracy rate and the manual sampling accuracy rate is greater than or equal to the first accuracy threshold, lowering the mine-burying proportion and / or raising the difficulty level of the mine file; when the first annotation accuracy rate and the manual sampling accuracy rate are both less than the second accuracy threshold, raising the mine-burying proportion and / or lowering the difficulty level of the mine file.

[0063] In some embodiments, the difficulty level of the Lei file is positively correlated with the first annotation accuracy or manual sampling accuracy of the data annotator. That is, the higher the first annotation accuracy or manual sampling accuracy of the data annotator, the higher the difficulty level of the Lei file used by the data annotator, and the lower the first annotation accuracy or manual sampling accuracy of the data annotator, the lower the difficulty level of the Lei file used by the data annotator. Among them, the difficulty level of the Lei file refers to the difficulty level of annotating the data to be annotated in the Lei file. The higher the difficulty level of the Lei file, the more difficult it is for the data annotator to annotate the data to be annotated in the Lei file, and the lower the difficulty level of the Lei file, the easier it is for the data annotator to annotate the data to be annotated in the Lei file.

[0064] Step S106: multiply the mine-laying ratio by the number of first data to be labeled in the data labeling task of the data labeler to obtain the target number of mine files.

[0065] For example, if the number of the first data to be labeled in the data labeling task of the data labeler is 1000 and the mine-laying ratio is 20%, then the target number of mine files is 1000*20%=200. For another example, if the number of the first data to be labeled in the data labeling task of the data labeler is 5000 and the mine-laying ratio is 8%, then the target number of mine files is 5000*8%=400. For another example, if the number of the first data to be labeled in the data labeling task of the data labeler is 3000 and the mine-laying ratio is 15%, then the target number of mine files is 3000*15%=450.

[0066] Step S107: Obtain a target number of thunder files from the thunder file pool, and add the obtained target number of thunder files to the data labeling task of the data labeler to update the data labeling task of the data labeler.

[0067] This embodiment obtains the mine-burying ratio corresponding to the data annotator, and according to the mine-burying ratio, obtains the corresponding mine file from the mine file pool, and adds the obtained mine file to the data annotator's data annotation task to update the data annotator's data annotation task, thereby ensuring that the mine-burying ratio of the mine file inserted into the data annotation task matches that of the data annotator, making it more convenient to perform quality inspection on the labeled data to be inspected obtained by the data annotator in the subsequent process, thereby improving the efficiency of quality inspection.

[0068] In some embodiments, a plurality of Lei files are stored in the Lei file pool, and the classification labels and / or difficulty levels of the plurality of Lei files may be the same or different, and the embodiments of the present invention do not specifically limit this. Among them, the classification labels of Lei files may include language labels, field labels, and business labels, etc. Language labels may include Chinese, English, Chinese and English mixed reading, Japanese, Korean, Russian, French, Arabic, and dialects, etc. Field labels may include general fields, medical fields, judicial fields, and security fields, etc. Business labels may include speech transcription, speech segmentation, picture classification, image semantic segmentation, OCR recognition, 3D point cloud, face and human skeleton dotting, and natural language processing annotation, etc.

[0069] In some embodiments, obtaining a target number of Lei files from a Lei file pool may include: determining the difficulty level and / or classification label of the data annotation task of the data annotator; obtaining a target number of Lei files from the Lei file pool according to the difficulty level and / or classification label of the data annotation task, wherein the difficulty level of the Lei files obtained matches the difficulty level of the data annotation task and / or the classification label of the Lei files obtained matches the classification label of the data annotation task. This embodiment implements adding Lei files that match the difficulty level and / or classification label of the data annotation task to the data annotation task, so that the added Lei files are more accurate, effectively improving the accuracy of the quality inspection of the annotated data, and the difficulty level of the Lei files can be gradually increased as the difficulty level of the data annotation task increases, which can enhance the comprehensiveness of the detection.

[0070] In some embodiments, the number of first data to be annotated between each two adjacent Lei files in the data annotation task is randomly determined; and or, each Lei file included in each data annotation task of the data annotator is different. This embodiment can prevent data annotators from cheating, make the labeled data to be inspected more real, and ensure the accuracy of the quality inspection of the labeled data. For example, by randomly determining the number of first data to be annotated between each two adjacent Lei files in the data annotation task, the position of the inserted Lei file in the data annotation task is unpredictable, which can effectively prevent the data annotator from inferring the appearance pattern of the Lei file through memory, thereby preventing the data annotator from cheating. For another example, by ensuring that each Lei file included in each data annotation task of the data annotator is different, it can be avoided that the Lei file repeatedly appears in front of the data annotator, resulting in the repeated Lei file being memorized by the data annotator, thereby preventing the data annotator from cheating.

[0071] In some embodiments, Figure 5 As shown, before step S105, the following steps are also included: Step S108: Obtain the thunder file uploaded by the user, and store the thunder file uploaded by the user in a thunder file pool.

[0072] This embodiment supports users to upload pre-prepared thunder files to the thunder file pool, so as to facilitate the subsequent extraction of thunder files of corresponding proportions from the thunder file pool for insertion when assigning data annotation tasks to data annotators or when data annotators receive data annotation tasks. It can be understood that step S108 can also be performed after step S105, or before step S101, step S102, step S103, step S104, step S106 or step S107, or after step S101, step S102, step S103, step S104, step S106 or step S107, and the embodiment of the present invention does not specifically limit this.

[0073] In some embodiments, the quality inspection method provided by the embodiment of the present invention also includes: determining the difficulty level and / or classification label of the Lei file uploaded by the user. Among them, the classification label of the Lei file can be obtained by automatically classifying the Lei file through a pre-trained classification model, or by manually classifying the Lei file by the user, and the embodiment of the present invention does not make specific limitations on this. The difficulty level of the Lei file can be obtained by automatically grading the difficulty of the Lei file through a pre-trained difficulty grading model, or by manually grading the difficulty of the Lei file by the user, and the embodiment of the present invention does not make specific limitations on this. This embodiment determines the difficulty level and / or classification label of the Lei file when the Lei file is uploaded, so as to facilitate the subsequent extraction of Lei files that match the difficulty level and / or classification label of the data annotation task from the Lei file pool when the data annotation task is assigned to the data annotator or the data annotator receives the data annotation task.

[0074] In some embodiments, the Lei file in the data labeling task is obtained from the Lei file pool, and the quality inspection method provided by the embodiment of the present invention also includes: in response to the fact that the delisting evaluation parameter of at least one Lei file in the Lei file pool meets the preset conditions, deleting at least one Lei file from the Lei file pool. Among them, the delisting evaluation parameter of at least one Lei file includes at least one of the number of times N of use of at least one Lei file, N single labeling consistency rates and average labeling consistency rate, N is an integer greater than or equal to 0, the single labeling consistency rate is determined based on the standard labeling result of the second data to be labeled in the Lei file and the inspection labeling result of the second data to be labeled, and the average labeling consistency rate is determined based on N single labeling consistency rates and the number of uses N. This embodiment can ensure the validity of the Lei files in the Lei file pool by deleting the Lei files whose delisting evaluation parameters meet the preset conditions in the Lei file pool, thereby improving the accuracy of quality inspection.

[0075] In some embodiments, the delisting evaluation parameters of at least one Lei file satisfying preset conditions include at least one of the following: the number of times at least one Lei file is used is greater than or equal to the preset number threshold; the average annotation consistency rate of at least one Lei file is greater than or equal to the first threshold; the number of single annotation consistency rates greater than or equal to the second threshold among the N single annotation consistency rates of at least one Lei file is greater than or equal to the first number threshold. Among them, the preset number threshold, the first threshold, the second threshold and the first number threshold can be set based on actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset number threshold is 5 times, the first threshold is 95%, the second threshold is 60%, and the first number threshold is 10 or 15.

[0076] In some embodiments, the delisting evaluation parameters of at least one thunder file satisfying preset conditions include at least one of the following: the number of times at least one thunder file is used is greater than or equal to the preset number threshold; the average annotation consistency rate of at least one thunder file is greater than or equal to the first threshold; the number of single annotation consistency rates of N single annotation consistency rates of at least one thunder file that are less than the third threshold is greater than or equal to the second number threshold. Among them, the preset number threshold, the first threshold, the second threshold and the second number threshold can be set based on actual conditions, and the embodiments of the present invention do not specifically limit this. For example, the preset number threshold is 6 times, the first threshold is 98%, the second threshold is 75%, and the second number threshold is 8 or 12.

[0077] For example, in response to the use times of at least one Lei file in the Lei file pool being greater than a preset number threshold, at least one Lei file is deleted from the Lei file pool, that is, Lei files with too many uses are deleted, so as to avoid the data labelers from remembering the repeatedly used ones, thereby preventing the data labelers from cheating, making the labeled data to be inspected more authentic, and ensuring the accuracy of the quality inspection of the labeled data. For example, in response to the average labeling consistency rate of at least one Lei file in the Lei file pool being greater than or equal to the first threshold, at least one Lei file is deleted from the Lei file pool, that is, overly simple Lei files are deleted, so as to avoid the use of overly simple Lei files for quality inspection, so as to ensure the accuracy of quality inspection.

[0078] For example, in response to the number of single-time annotation consistency rates greater than or equal to the second threshold value among the N single-time annotation consistency rates of at least one thunder file in the thunder file pool being greater than or equal to the first numerical threshold value, at least one thunder file is deleted from the thunder file pool, that is, the thunder file that is too simple is deleted, so as to ensure the difficulty of the thunder files in the thunder file pool, so as to avoid using too simple thunder files for quality inspection, so as to ensure the accuracy of the quality inspection. For example, in response to the number of single-time annotation consistency rates less than the third threshold value among the N single-time annotation consistency rates of at least one thunder file in the thunder file pool being greater than or equal to the second numerical threshold value, at least one thunder file is deleted from the thunder file pool, that is, the very difficult thunder file is deleted, so as to ensure the difficulty of the thunder files in the thunder file pool, so as to avoid using very difficult thunder files for quality inspection, so as to ensure the efficiency of the quality inspection.

[0079] In some embodiments, after step S104, it also includes: when the quality inspection result is that the labeled data to be inspected passes the quality inspection, the Lei file is deleted from the labeled data to be inspected to obtain the initial inspection labeled data; after determining that the initial inspection labeled data passes the manual sampling inspection or manual verification, each first data to be marked in the initial inspection labeled data and the corresponding labeling result to be inspected are stored in the Lei file pool as a new Lei file. Wherein, manual sampling refers to: when the quality inspection result is that the labeled data to be inspected passes the quality inspection, and the labeled data to be inspected is not submitted for the first time, the quality inspector continues to perform manual sampling inspection on the labeled data to be inspected. Manual verification refers to: when the quality inspection result is that the labeled data to be inspected passes the quality inspection, and the labeled data to be inspected is submitted for the first time, the manual sampling inspection is exempted, and in order to ensure the accuracy of the Lei files returned to the Lei file pool, the quality inspector performs manual verification on the labeled data to be inspected. This embodiment can automatically return the qualified labeled data to the Lei file pool, so as to realize the iterative update of the Lei file pool, so as to ensure the accuracy of the Lei file, thereby ensuring the accuracy when using the Lei file for quality inspection.

[0080] For example, Figure 6As shown, the user uploads the prepared thunder file to the server in advance, and the server stores the thunder file in the thunder file pool. The server can determine the classification label and / or difficulty level of the thunder file. The classification labels include: different languages ​​(such as Chinese, English, Chinese and English mixed reading, dialects, etc.), different fields (such as general, medical, judicial, security, etc.), and different businesses (such as speech transcription, speech segmentation, image classification, image semantic segmentation, OCR recognition, 3D point cloud, face and human skeleton dotting, NLP annotation, etc.). When the data annotator receives or is assigned a data annotation task, the server extracts a certain proportion of thunder files from the thunder file pool to the data to be annotated.

[0081] In the labeling process, the data annotator labels the labeled data containing the Lei file, and then submits the labeled data to be inspected containing the Lei file to the server. In the quality inspection process, the server automatically performs quality inspection on the labeled data to be inspected containing the Lei file: the server determines the first labeling accuracy of the Lei file based on the consistency rate between the standard labeling result of the second data to be labeled in the Lei file and the labeling result to be inspected of the same second data to be labeled. If the first labeling accuracy is less than the preset accuracy threshold, it will be automatically returned to the data annotator; if the first labeling accuracy is greater than or equal to the preset accuracy threshold, it is determined whether it is the first submission. If the data annotator does not submit the labeled data to be inspected containing the Lei file for the first time (i.e., it has been rejected), it will enter the manual sampling stage (the Lei file will not be sampled in this stage of sampling). After passing the manual sampling, the qualified labeled data that has passed the manual sampling can be returned to the Lei file pool. If the data annotator submits the labeled data to be inspected containing the Lei file for the first time, it will be automatically exempted from inspection, i.e., exempted from manual sampling. Among them, the strategy of exempting manual sampling can be turned off. If the strategy of exempting manual sampling is turned off, the manual sampling stage will be entered (this stage of sampling will not extract thunder files). After passing the manual sampling, the qualified annotated data that has passed the manual sampling can be returned to the thunder file pool. For example, Figure 7 As shown, after the manual sampling is exempted, the quality inspectors can continue to manually verify the labeled data to be inspected, and then return the qualified labeled data that have passed the manual verification to the Lei file pool.

[0082] See also Figure 8 , Figure 8 It is a schematic block diagram of the structure of a server provided by an embodiment of the present invention.

[0083] like Figure 8 As shown, the server 100 includes a processor 101 and a memory 102 , and the processor 101 and the memory 102 are connected via a bus 103 , such as an I2C (Inter-integrated Circuit) bus.

[0084] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire server. The processor 101 can be a central processing unit (CPU), and the processor 101 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0085] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0086] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the server to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] The processor 101 is used to run a computer program stored in the memory 102, and implement any one of the quality inspection methods for labeled data provided by the embodiments of the present invention when executing the computer program.

[0088] In some embodiments, the processor 101 is used to run a computer program stored in the memory, and implement the following steps when executing the computer program: Acquire the labeled data to be inspected containing a Lei file, wherein the labeled data to be inspected is generated by a data annotator after completing a data labeling task containing the Lei file and the first data to be labeled, the Lei file includes the second data to be labeled and the standard labeling result of the second data to be labeled, and the labeled data to be inspected includes the labeling result to be inspected of the second data to be labeled; Determine a consistency rate between the standard annotation result of the second data to be annotated in the thunder file and the to-be-checked annotation result of the same second data to be annotated; Determining a first annotation accuracy rate of the data annotation task according to the consistency rate; A quality inspection result of the labeled data to be inspected is determined according to the first annotation accuracy rate.

[0089] In some embodiments, before obtaining the labeled data to be inspected containing the thunder file, the processor 101 is further configured to implement: Obtaining a mine-laying ratio corresponding to the data annotator, where the mine-laying ratio is a ratio of the number of mine files inserted into the data annotation task to the number of first data to be annotated in the data annotation task; Multiplying the mine-laying ratio by the number of first data to be annotated in the data annotation task of the data annotator to obtain the target number of mine files; The target number of thunder files are obtained from the thunder file pool, and the obtained target number of thunder files are added to the data labeling task of the data labeler to update the data labeling task of the data labeler.

[0090] In some embodiments, when obtaining the mine-laying ratio corresponding to the data labeler, the processor 101 is used to implement: Obtaining a second annotation accuracy rate of the data annotator, where the second annotation accuracy rate is determined according to a first annotation accuracy rate of each data annotation task completed by the data annotator; According to the second labeling accuracy rate, the mine-laying ratio corresponding to the data labeler is determined.

[0091] In some embodiments, after determining the first annotation accuracy rate of the data annotation task according to the consistency rate, the processor 101 is further configured to implement: According to the first annotation accuracy rate of the data annotation task, adjust the mine-burying ratio and / or the difficulty level of the mine file; and / or, according to the manual sampling accuracy rate of the data annotation task, adjust the mine-burying ratio and / or the difficulty level of the mine file, the manual sampling accuracy rate is determined based on the sampling annotation results obtained by the quality inspector on part of the first data to be labeled in the labeled data to be inspected corresponding to the data annotation task and the labeling results to be inspected of the part of the first data to be labeled.

[0092] In some embodiments, when the processor 101 obtains the target number of thunder files from the thunder file pool, it is used to implement: Determining a difficulty level and / or classification label of the data labeling task of the data labeler; According to the difficulty level and / or classification label of the data labeling task, the target number of thunder files are obtained from the thunder file pool, wherein the difficulty level of the obtained thunder files matches the difficulty level of the data labeling task and / or the classification label of the obtained thunder files matches the classification label of the data labeling task.

[0093] In some embodiments, the number of the first data to be annotated between each two adjacent thunder files in the data annotation task is randomly determined; and / or, the thunder files included in each data annotation task of the data annotator are different.

[0094] In some embodiments, the thunder file in the data annotation task is obtained from a thunder file pool, and the processor 101 is further configured to implement: In response to a delisting evaluation parameter of at least one of the thunder file in the thunder file pool meeting a preset condition, the at least one thunder file is deleted from the thunder file pool.

[0095] In some embodiments, the delisting evaluation parameter of the at least one thunder file includes at least one of the number of times N of use of the at least one thunder file, N single annotation consistency rates, and an average annotation consistency rate, where N is an integer greater than or equal to 0, the single annotation consistency rate is determined according to the standard annotation result of the second data to be annotated in the thunder file and the inspection annotation result of the second data to be annotated, and the average annotation consistency rate is determined according to the N single annotation consistency rates and the number of times N of use; The delisting evaluation parameters of the at least one Lei file satisfying the preset conditions include at least one of the following: the number of times the at least one Lei file is used is greater than or equal to the preset number threshold; the average annotation consistency rate of the at least one Lei file is greater than or equal to the first threshold; the number of the single annotation consistency rates of the N single annotation consistency rates of the at least one Lei file that are greater than or equal to the second threshold is greater than or equal to the first number threshold; Alternatively, the delisting evaluation parameters of the at least one thunder file satisfying preset conditions include at least one of the following: the number of times the at least one thunder file is used is greater than or equal to a preset number threshold; the average annotation consistency rate of the at least one thunder file is greater than or equal to a first threshold; the number of the single annotation consistency rates of the N single annotation consistency rates of the at least one thunder file that are less than a third threshold is greater than or equal to a second numerical threshold.

[0096] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the server described above can refer to the corresponding process in the aforementioned quality inspection method embodiment of labeled data, and will not be repeated here.

[0097] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any quality inspection method for labeled data provided in the description of the embodiment of the present invention.

[0098] The storage medium may be an internal storage unit of the server described in the foregoing embodiment, such as a hard disk or memory of the server. The storage medium may also be an external storage device of the server, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the server.

[0099] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0100] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0101] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A quality inspection method for labeled data, characterized in that: include: Acquire the labeled data to be inspected containing a Lei file, wherein the labeled data to be inspected is generated by a data annotator after completing a data labeling task containing the Lei file and the first data to be labeled, the Lei file includes the second data to be labeled and the standard labeling result of the second data to be labeled, and the labeled data to be inspected includes the labeling result to be inspected of the second data to be labeled; Determine a consistency rate between the standard annotation result of the second data to be annotated in the thunder file and the to-be-checked annotation result of the same second data to be annotated; A first annotation accuracy rate of the data annotation task is determined according to the consistency rate, and a quality inspection result of the to-be-inspected annotated data is determined according to the first annotation accuracy rate.

2. The quality inspection method according to claim 1, characterized in that: Before obtaining the labeled data to be inspected containing the mine file, the method further includes: Obtaining a mine-laying ratio corresponding to the data annotator, where the mine-laying ratio is a ratio of the number of mine files inserted into the data annotation task to the number of first data to be annotated in the data annotation task; Multiplying the mine-laying ratio by the number of first data to be annotated in the data annotation task of the data annotator to obtain the target number of mine files; The target number of thunder files are obtained from the thunder file pool, and the obtained target number of thunder files are added to the data labeling task of the data labeler to update the data labeling task of the data labeler.

3. The quality inspection method according to claim 2, characterized in that: The obtaining the mine-laying ratio corresponding to the data labeler includes: Obtaining a second annotation accuracy rate of the data annotator, where the second annotation accuracy rate is determined according to a first annotation accuracy rate of each data annotation task completed by the data annotator; According to the second labeling accuracy rate, the mine-laying ratio corresponding to the data labeler is determined.

4. The quality inspection method according to claim 2, characterized in that: After determining the first annotation accuracy rate of the data annotation task according to the consistency rate, the method further includes: According to the first annotation accuracy rate of the data annotation task, adjust the mine-burying ratio and / or the difficulty level of the mine file; and / or, according to the manual sampling accuracy rate of the data annotation task, adjust the mine-burying ratio and / or the difficulty level of the mine file, the manual sampling accuracy rate is determined based on the sampling annotation results obtained by the quality inspector on part of the first data to be labeled in the labeled data to be inspected corresponding to the data annotation task and the labeling results to be inspected of the part of the first data to be labeled.

5. The quality inspection method according to claim 2, characterized in that: The step of obtaining the target number of mine files from the mine file pool includes: Determining a difficulty level and / or classification label of the data labeling task of the data labeler; According to the difficulty level and / or classification label of the data labeling task, the target number of thunder files are obtained from the thunder file pool, wherein the difficulty level of the obtained thunder files matches the difficulty level of the data labeling task and / or the classification label of the obtained thunder files matches the classification label of the data labeling task.

6. The quality inspection method according to claim 2, characterized in that: The number of the first to-be-annotated data between each two adjacent thunder files in the data annotation task is randomly determined; and / or, the thunder files included in each data annotation task of the data annotator are different.

7. The quality inspection method according to any one of claims 1 to 6, characterized in that: The thunder file in the data annotation task is obtained from a thunder file pool, and the method further includes: In response to a delisting evaluation parameter of at least one of the thunder file in the thunder file pool meeting a preset condition, the at least one thunder file is deleted from the thunder file pool.

8. The quality inspection method according to claim 7, characterized in that: The delisting evaluation parameter of the at least one thunder file includes at least one of the number of times N of use of the at least one thunder file, N single annotation consistency rates, and an average annotation consistency rate, where N is an integer greater than or equal to 0, the single annotation consistency rate is determined according to the standard annotation result of the second data to be annotated in the thunder file and the inspection annotation result of the second data to be annotated, and the average annotation consistency rate is determined according to the N single annotation consistency rates and the number of times N of use; The delisting evaluation parameters of the at least one Lei file satisfying the preset conditions include at least one of the following: the number of times the at least one Lei file is used is greater than or equal to the preset number threshold; the average annotation consistency rate of the at least one Lei file is greater than or equal to the first threshold; the number of the single annotation consistency rates of the N single annotation consistency rates of the at least one Lei file that are greater than or equal to the second threshold is greater than or equal to the first number threshold; Alternatively, the delisting evaluation parameters of the at least one thunder file satisfying preset conditions include at least one of the following: the number of times the at least one thunder file is used is greater than or equal to a preset number threshold; the average annotation consistency rate of the at least one thunder file is greater than or equal to a first threshold; the number of the single annotation consistency rates of the N single annotation consistency rates of the at least one thunder file that are less than a third threshold is greater than or equal to a second numerical threshold.

9. A server, characterized in that: The server includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the quality inspection method for labeled data according to any one of claims 1 to 8 is realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the quality inspection method for labeled data according to any one of claims 1 to 8.

Images