Label data quality control method in multi-person cooperation scene

Abstract

A labeling data quality control method under a multi-person cooperation scene comprises the steps that samples are selected from a to-be-labeled data set, then labeling personnel label the selected samples to obtain sample labeling results, the data set is labeled with reference to the samples, the accuracy rate, the recall rate and the F1 score of the labeling result of each labeling personnel are calculated, and the accuracy rate, the recall rate and the F1 score of the labeling result of each labeling personnel are calculated; according to the accuracy rate, the recall rate and the F1 score, the labeling personnel class is divided into qualification labeling personnel and general labeling personnel, and task allocation is dynamically performed on the qualification labeling personnel and the general labeling personnel; by the adoption of the method, the problem that attention of annotation personnel is reduced in the continuous annotation process is solved, so that the annotation quality is improved, and the error rate and the rework cost are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing (NLP), and in particular to a method for controlling the quality of labeled data in a multi-person collaboration scenario. Background Art

[0002] As natural language processing applications such as intelligent robots, public opinion analysis, and automatic summarization become more and more widespread, the demand for labeled data is also increasing. One person or a few people alone cannot complete the labeling task within a reasonable time. The existing multi-person collaborative labeling solution can make full use of group wisdom and significantly improve the efficiency of data labeling.

[0003] To ensure the quality of the labeled data, it is necessary to apply reasonable and effective quality control methods to improve the reliability of the labeled data. A complete data labeling process is divided into four stages: requirements understanding, data acquisition, data labeling, and data acceptance. The existing technical solutions start to control the labeling quality from the stage of understanding the project's requirements, and perform quality control in multiple stages such as personnel training, self-inspection, cross-checking, and quality review. Only after the final quality inspection is correct, the collaborative labeling data is formally accepted. The methods adopted in each stage include:

[0004] (1) In the demand understanding stage, correctly understand the annotation requirements, annotation standards, sample quantity and required time, etc., formulate unified annotation specifications and standard operating procedures based on important factors such as data source, format, requirements and data analysis purpose, and explain and interpret them for subsequent reference or training of annotation personnel.

[0005] (2) In the early preparation stage, the annotation team is trained on the annotation requirements, annotation methods, annotation methods, and the use of annotation software obtained in the demand understanding stage, so that all annotation personnel can understand and master the annotation requirements and annotation standards, be familiar with the characteristics of the annotated data, and use the correct annotation methods and tools to carry out data annotation tasks;

[0006] (3) In the data labeling stage, on the basis that the labelers can use the correct labeling methods and tools. First, after reviewing the labeling tasks they are responsible for, each labeler will check the labeling results of other labelers and promptly correct their own labeling errors to minimize subjective bias; secondly, in the case of inconsistent labeling results, the final labeling results will be arbitrated by the labeling reviewers to further ensure the consistency and correctness of the data labeling; finally, the person in charge or the labeling reviewer will randomly check the labeling results and rework the tasks that do not meet the labeling requirements.

[0007] (4) During the data acceptance stage, a certain proportion of the labeled data is extracted for quality inspection. If errors or inaccurate labeling results are found, clear correction instructions and relevant reasons are given. After confirming that the labeling quality meets the requirements, data acceptance is carried out.

[0008] However, current annotation systems and technical solutions do not fully consider the propagation and diffusion of annotation errors in the process of large-scale annotation, lack the precise matching between the current annotation tasks and personnel capabilities, and do not pay attention to the changes in the attention of the annotators throughout the annotation process. These problems lead to the spread of errors and affect the final annotation quality, as well as higher error rates and rework costs.

[0009] The Chinese patent application with publication number CN112766347A discloses an active learning method combined with annotation quality control. However, since the propagation and diffusion of annotation errors are not considered, the quality of the entire annotation project will eventually be adversely affected.

[0010] The Chinese patent application with publication number CN112749563A discloses a method and system for evaluating and controlling the quality of named entity recognition data annotation. However, since the attention changes of the annotators during the entire annotation process are not taken into account, the fluctuations in the annotation quality caused by the attention changes may lead to higher error rates and rework costs. Summary of the invention

[0011] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method for quality control of annotation data in a multi-person collaborative scenario, by introducing sample annotation before the formal annotation begins, testing whether each annotation person can understand and follow the same annotation standard, and dividing the personnel type according to the performance of each annotation person in sample annotation, so as to automatically assign data of varying difficulty to different annotation persons, and then during multi-person collaborative annotation, dynamically adjust the tasks according to the annotation quality feedback data collected in real time to prevent the problem of decreased attention of the annotation persons during the continuous annotation process, thereby improving the annotation quality and reducing the error rate and rework cost.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is:

[0013] A method for quality control of annotation data in a multi-person collaboration scenario includes the following steps:

[0014] Step 1: First, select samples from the dataset to be labeled, and then the labelers label the selected samples to obtain the sample labeling results. Referring to the sample labeling dataset, calculate the accuracy, recall rate, and F1 score of each labeler's labeling results;

[0015] Step 2: Based on the accuracy, recall, and F1 score obtained in step 1, the labelers are divided into senior labelers and general labelers.

[0016] Step 3: Dynamically assign tasks to the senior labelers and general labelers divided in step 2.

[0017] The step 1 specifically includes:

[0018] First, according to the pre-established annotation specifications and standard operating procedures, samples are selected from the dataset to be annotated for annotation, and a sample annotation dataset is produced;

[0019] The examples cover all label types and include annotation tasks of different difficulty levels to ensure the diversity of the sample data;

[0020] The label type is customized by the user according to different labeling tasks, taking the text labeling task of named entity recognition as an example;

[0021] The simple difficulty of the text annotation task of named entity recognition is to annotate a category of named entities in the text;

[0022] A medium-difficulty text annotation task for named entity recognition: annotating multiple categories of named entities;

[0023] Complex and difficult text annotation tasks for named entity recognition: annotating entities in long texts and solving the coreference resolution problem;

[0024] Next, all the labelers are trained to understand the labeling requirements, labeling methods and the use of labeling tools. Then the sample data is distributed to all the labelers and labeling instructions are provided. The labelers independently complete the labeling of the sample data and submit the results within the specified time.

[0025] The annotation instructions provide annotation specifications and operation procedures;

[0026] Finally, referring to the sample annotation dataset, we calculated the accuracy, recall, and F1 score of each annotator’s annotation results to assess whether the annotator has correctly understood and followed the annotation instructions:

[0027] Confusion Matrix in Classification Problems

[0028]

[0029] In the matrix, 1 represents the positive class and 0 represents the negative class;

[0030] TP (True Positives): When the actual result and the predicted result are both 1, it is a true class, indicating the number of positive classes correctly classified as positive classes;

[0031] TN (True Negative): When the actual result and the predicted result are both 0, it is a true negative class, indicating the number of negative classes that are correctly classified as negative classes;

[0032] FP (False Positive): When the actual result is 0 but the predicted result is 1, it is a false positive class, indicating the number of negative classes that are mistakenly classified as positive classes;

[0033] FN (False Negative): When the actual result is 1 but the predicted result is 0, it is a false negative class, indicating the number of positive classes that are mistakenly classified as negative classes;

[0034] The calculation method of accuracy is:

[0035] Accuracy is the ratio of correctly classified samples to the total number of samples, indicating the overall predictive ability of the model. The calculation formula is:

[0036]

[0037] The calculation method of recall rate is:

[0038] Recall, also known as recall rate, is the probability of being correctly predicted as a positive sample among samples that are actually positive. The calculation formula is:

[0039]

[0040] F1 score calculation method:

[0041] The F1 score is the harmonic mean of precision and recall, which combines the performance of the two. It strikes a balance between precision and recall and is suitable for situations where there is an imbalance between classes in classification tasks:

[0042]

[0043] Among them, precision is the proportion of samples that are actually positive among all samples predicted to be positive. This is an indicator to measure the accuracy of the results, especially for cases where positive prediction is concerned. The calculation formula is:

[0044]

[0045] The step 2 specifically includes:

[0046] The senior labelers are those with an accuracy rate higher than m% and an error rate lower than (100-m)%;

[0047] The general labelers are labelers whose accuracy is within the range of (n-m)% and whose error rate is within the range of (100-m)-(100-n)%;

[0048] Where 0≤n <m≤100。

[0049] The step 3 specifically includes:

[0050] The dynamic task adjustment of senior annotators in the actual annotation process is divided into the following steps:

[0051] Step 3.1: Use the A3C algorithm to track changes in the annotation quality of senior annotators during the annotation process, so as to dynamically capture fluctuations in attention levels. The annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of senior annotators;

[0052] Step 3.2: If the annotation quality of the senior annotator is lower than A as evaluated in step 3.1, the attention is reduced, and the type or content of the annotation task is changed to maintain high annotation quality. The value A is observed by the relationship between the annotation quality calculated by the person in charge or the expert through the A3C algorithm and the number of completed annotation tasks;

[0053] The dynamic task adjustment of general labelers is divided into the following steps:

[0054] Step S3.1: Track the changes in the annotation quality of general annotators during the annotation process through the A3C algorithm, so as to dynamically capture the fluctuation of the attention level, wherein the annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of the general annotators;

[0055] Step S3.2: If the annotation quality of the general annotator is lower than B as evaluated in step S3.1, the attention is reduced, and the task assignment is reduced in time or rest time is given, or incentives are used to improve attention. The B value is observed by the relationship between the annotation quality calculated by the A3C algorithm by the person in charge or the expert and the number of completed annotation tasks;

[0056] Step S3.3: When the annotation quality of general annotators is improved to higher than C through step S3.2, attention is increased, and the amount of tasks or task difficulty is increased, that is, specific fields or task types are expanded. The C value is observed by the relationship between the annotation quality calculated by the A3C algorithm by the person in charge or the expert and the number of completed annotation tasks;

[0057] The annotation speed in step 3.1 or step S3.1 refers to the amount of annotation tasks completed per unit time, which is used to measure the efficiency of the annotator or system. The specific calculation method varies according to different annotation tasks. The following is a general calculation method for annotation speed:

[0058]

[0059] Completed annotations: The number of annotation tasks completed by the annotator within the specified time.

[0060] Total time: The total time it takes for the annotator to complete these tasks, measured in hours, minutes, etc.

[0061] The accuracy in step 3.1 or step S3.1 is:

[0062]

[0063] Confusion matrix in classification problems:

[0064]

[0065] In the matrix, 1 represents the positive class and 0 represents the negative class;

[0066] TP (True Positives): When the actual result and the predicted result are both 1, it is a true class, indicating the number of positive classes correctly classified as positive classes;

[0067] TN (True Negative): When the actual result and the predicted result are both 0, it is a true negative class, indicating the number of negative classes that are correctly classified as negative classes;

[0068] FP (False Positive): When the actual result is 0 but the predicted result is 1, it is a false positive class, indicating the number of negative classes that are mistakenly classified as positive classes;

[0069] FN (False Negative): When the actual result is 1 but the predicted result is 0, it is a false negative class, indicating the number of positive classes that are mistakenly classified as negative classes.

[0070] The incentives described in step S3.2 include points or rewards;

[0071] The points specifically include: task completion points and high-quality annotation points;

[0072] The rewards specifically include: money, honor, and gifts.

[0073] The A3C algorithm (Asynchronous Advantage Actor-Critic) described in the above steps is a policy gradient-based reinforcement learning method that achieves fast and stable learning results through asynchronous training of multiple agents.

[0074] The training process of the A3C algorithm is divided into the following steps:

[0075] (1) Initialize neural network parameters;

[0076] (2) Create multiple parallel training threads, each of which independently runs an agent to interact with the environment and uses the Actor and Critic network to approximate the strategy and value;

[0077] (3) Each thread selects an action based on the current policy network and observes the new state and reward, storing this information in the experience replay buffer;

[0078] (4) When a thread reaches a certain number of time steps or the trajectory ends, the thread samples the data in the experience replay buffer and performs gradient updates by calculating the advantage function;

[0079] (5) After each thread performs a certain number of gradient updates, it passes the updated parameters to the main thread for overall parameter update;

[0080] (6) Repeat the above steps until the predetermined training rounds are reached or the termination condition is met.

[0081] Compared with the prior art, the beneficial effects of the present invention are:

[0082] 1. Higher labeling consistency and more stable labeling quality: By introducing a sample labeling mechanism, labelers are tested and evaluated before the formal labeling begins. Sample data is assigned to all labelers and detailed labeling instructions are provided. They are required to independently complete the labeling of sample data within a specified time and submit the results for testing. At the same time, the accuracy, recall rate, and F1 score of each labeler's labeling results are calculated to evaluate whether all personnel correctly understand and follow the unified labeling standards. This mechanism helps reduce errors and deviations in the labeling process and improves the consistency and accuracy of the overall labeling. Existing technologies often lack such early standardized testing methods.

[0083] 2. Personalized labeling task allocation: The present invention dynamically adjusts task allocation and workload based on the real-time performance of labelers, and especially designs refined task adjustment strategies for individual differences of different types of labelers. This flexible allocation method significantly reduces the fluctuation of labeling quality caused by decreased attention, and ensures the labeling efficiency for long-term work. In contrast, the existing technology uses a fixed task allocation method, which cannot effectively cope with the changes in labelers' attention, and thus easily leads to the problem of decreased labeling quality.

[0084] In summary, compared with the prior art, the present invention reduces errors and deviations in the annotation process, improves the consistency and accuracy of the overall annotation, significantly reduces the fluctuation of annotation quality caused by decreased attention, and ensures the annotation efficiency for long-term work. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 The figure is a schematic diagram of the entire data annotation process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0087] like Figure 1 As shown, the entire data labeling process is divided into four stages: demand understanding, preliminary preparation, data labeling, and data acceptance.

[0088] Phase 1: Requirements understanding. The person in charge or expert first understands the annotation requirements, then determines the specific content of the annotation task, then clarifies the different annotation types, and finally formulates the corresponding annotation standards;

[0089] Phase 2: Preliminary preparation. According to the content of the annotation task in the first phase, collect the original data that needs to be annotated, pre-process the original data to make it the data to be annotated, and select representative small-scale samples from the complete data set to be annotated by experts to produce a sample annotation data set and obtain an annotation benchmark for subsequent reference;

[0090] Based on the annotation types and standards in the first phase, formulate annotation specifications, select appropriate annotation software, train an annotation team based on the annotation specifications and software, and distribute small-scale sample data to the annotation team, who will annotate the samples. Finally, refer to the standard benchmark to obtain the annotation results of each annotator.

[0091] Phase 3: Data labeling. Before the formal labeling, by collecting the labeling results of each labeler in the second phase, we evaluate their understanding of the current labeling standards and their ability to handle tasks of different difficulty levels, thereby dividing them into senior labelers and general labelers. In the entire labeling process, the type of labeler is not static, but is obtained by real-time evaluation using the continuously collected labeling results of multiple people. The global labeling quality of senior labelers is very high. They can easily adapt to different labeling tasks and their attention on a given task changes little. For general labelers, the attention of labelers will fluctuate over time. Therefore, the labeling speed, accuracy and other indicators of each labeler are monitored in real time, and their attention level is dynamically evaluated. When the labeler's attention decreases or increases, the data to be labeled is reallocated to achieve dynamic task allocation. When problems are encountered during the labeling process, they are handled by the responsible person or expert.

[0092] The fourth stage: data acceptance. The multi-person labeling results annotated by the general labelers in the third stage are obtained. If the acceptance is passed, the accepted labeling results are output to obtain the final labeling results.

[0093] A method for quality control of annotation data in a multi-person collaboration scenario includes the following steps:

[0094] Step 1: First, select samples from the dataset to be labeled, and then the labelers label the selected samples to obtain the sample labeling results. Referring to the sample labeling dataset, calculate the accuracy, recall rate, and F1 score of each labeler's labeling results;

[0095] Step 2: Based on the accuracy, recall, and F1 score obtained in step 1, the labelers are divided into senior labelers and general labelers.

[0096] Step 3: Dynamically assign tasks to the senior labelers and general labelers divided in step 2.

[0097] The step 1 specifically includes:

[0098] Before starting the small batch sample annotation, the person in charge or the expert first selects representative small-scale samples (such as 10-20 data items) from the complete dataset to be annotated according to the pre-established annotation specifications and standard operating procedures. In this embodiment, 20 data items are used for annotation to produce a sample annotation dataset.

[0099] When selecting samples, it should be noted that the samples cover all label types and include annotation tasks of different difficulty levels to ensure the diversity of sample data;

[0100] The label type is customized by the user according to different labeling tasks. Taking the text labeling task of named entity recognition as an example, the label types include: name, organization, place, time, product, event, and labeling tasks of different difficulty levels are divided into: simple, medium, and complex;

[0101] Simple difficulty annotation task of named entity recognition text annotation task: annotate one category of named entities in the text, such as annotating only people's names;

[0102] A medium-difficulty text annotation task for named entity recognition: annotating multiple categories of named entities, such as annotating names of people, places, and organizations at the same time;

[0103] Complex and difficult text annotation tasks for named entity recognition: annotating entities in long texts and solving the coreference resolution problem;

[0104] Taking image annotation in the autonomous driving scenario as an example, the label types include: vehicles, pedestrians, and traffic signs. The annotation tasks of different difficulty levels are divided into: simple, medium, and complex;

[0105] Simple and difficult image annotation tasks in autonomous driving scenarios: The target object is clearly visible, the environment is simple, and the annotation boundary is clear. Usually only a small number of objects are involved. For example, there is a car driving on an empty street in the image with no other interference. The car in the image is labeled as a vehicle and annotated with a rectangular box.

[0106] Medium-difficulty image annotation tasks in autonomous driving scenarios: There are multiple target objects in the scene, and the target objects may be partially occluded and the environment is slightly complex, but the target objects are easy to distinguish. For example, in an image, there are multiple cars driving on a street and pedestrians walking on the sidewalk. Label each car and pedestrian separately and mark the bounding box.

[0107] Complex and difficult image annotation tasks in autonomous driving scenarios: There are many and complex target objects, and there may be heavy occlusion, poor lighting conditions, blurred target objects, or dynamic changes. For example, the image has poor lighting at night, and cars and pedestrians on the street are in weak light, and some pedestrians pass through shadow areas. In low-light conditions, mark the positions of cars and pedestrians and label them as vehicles and pedestrians;

[0108] Next, a brief training is given to all the annotators to make them understand the annotation requirements, annotation methods and the use of annotation tools. Then, sample data is assigned to all the annotators and detailed annotation instructions are provided. The annotators are required to independently complete the annotation of the sample data and submit the results within the specified time.

[0109] The annotation instructions provide the annotation specifications and operating procedures used by the responsible person / senior annotator when annotating the samples;

[0110] Finally, referring to the sample annotation dataset, we calculated the accuracy, recall, and F1 score of each annotator’s annotation results to assess whether the annotator has correctly understood and followed the annotation instructions:

[0111] Confusion Matrix in Classification Problems

[0112]

[0113] In the matrix, 1 represents the positive class and 0 represents the negative class;

[0114] TP (True Positives): When the actual result and the predicted result are both 1, it is a true class, indicating the number of positive classes correctly classified as positive classes;

[0115] TN (True Negative): When the actual result and the predicted result are both 0, it is a true negative class, indicating the number of negative classes that are correctly classified as negative classes;

[0116] FP (False Positive): When the actual result is 0 but the predicted result is 1, it is a false positive class, indicating the number of negative classes that are mistakenly classified as positive classes;

[0117] FN (False Negative): When the actual result is 1 but the predicted result is 0, it is a false negative class, indicating the number of positive classes that are mistakenly classified as negative classes;

[0118] The calculation method of accuracy is:

[0119] Accuracy is the ratio of correctly classified samples to the total number of samples, indicating the overall predictive ability of the model. The calculation formula is:

[0120]

[0121] The calculation method of recall rate is:

[0122] Recall, also known as recall rate, is the probability of being correctly predicted as a positive sample among samples that are actually positive. The calculation formula is:

[0123]

[0124] F1 score calculation method:

[0125] The F1 score is the harmonic mean of precision and recall, which combines the performance of the two. It strikes a balance between precision and recall and is suitable for situations where there is an imbalance between classes in classification tasks:

[0126]

[0127] Among them, precision is the proportion of samples that are actually positive among all samples predicted to be positive. This is an indicator to measure the accuracy of the results, especially for cases where positive prediction is concerned. The calculation formula is:

[0128]

[0129] Taking the text annotation NER task as an example, when the annotation performance of all annotators is generally poor, that is, when >50% of the annotators have an annotation result with an accuracy of <80%, an F1 score of <75%, or a recall rate of <75%, the annotation instructions need to be readjusted to ensure that they are clearer and easier to understand. When the annotation result of annotators has an accuracy of <75%, an F1 score of <70%, or a recall rate of <70%, further training needs to be provided to provide feedback on the evaluation results to the annotators, pointing out their errors or inconsistencies.

[0130] The step 2 specifically includes:

[0131] Individual differences can significantly affect the overall consistency of multi-person annotation. The division of annotation personnel that matches tasks with their abilities can avoid annotation errors caused by assigning tasks that are too difficult, and also avoid efficiency waste caused by tasks that are too simple. This can not only improve the job satisfaction of each annotation personnel, but also greatly improve the overall quality of the entire annotation project. It is known that annotation personnel are mainly divided into two categories: senior annotation personnel and general annotation personnel. Senior annotation personnel refer to annotation personnel with rich annotation experience, stable quality, and low error rate, who are more suitable for completing review and proofreading work; general annotation personnel refer to annotators with certain experience, but with fluctuating annotation quality. They may not be cautious enough when annotating, and disputes are prone to occur between each other, and their annotation data requires a certain degree of quality inspection.

[0132] Before the formal labeling, by collecting the performance of each labeler in small-scale sample labeling, we evaluate their understanding of the current labeling standards and their ability to handle tasks of different difficulty levels. This allows senior labelers and general labelers to be divided. Labelers with excellent performance are classified as senior labelers, while labelers with average performance are classified as general labelers. Based on this, personalized initial task allocation enables each labeler to start with tasks that suit their ability level. Throughout the labeling process, the type of labeler is not static, but is obtained through real-time evaluation using the continuously collected labeling data.

[0133] Work experience can be measured by the annotator’s cumulative working hours, the number of tasks completed, or the number of projects participated in.

[0134] The quality of annotation can be measured by real-time monitoring of the annotation speed and accuracy obtained by experienced annotators.

[0135] The error rate is usually expressed as the percentage of incorrect annotations relative to the total number of annotation tasks.

[0136] Taking the text annotation NER task as an example, the senior annotator refers to an annotator with an accuracy rate higher than 90% and an error rate lower than 10%;

[0137] The general labeling personnel are labeling personnel whose accuracy rate is within the range of 80-90% and whose error rate is within the range of 10-20%.

[0138] The step 3 specifically includes:

[0139] The global annotation quality of experienced annotators is very high. They can easily adapt to different annotation tasks and have little variation in their attention on a given task.

[0140] Senior labelers can adjust tasks dynamically during the actual labeling process in the following steps:

[0141] Step 3.1: Use the A3C algorithm to track changes in the annotation quality of the annotators during the annotation process, so as to dynamically capture fluctuations in attention levels. The annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of senior annotators;

[0142] Step 3.2: If the annotation quality of the senior annotator is lower than A as evaluated in step 3.1, the attention is reduced, and the type or content of the annotation task is changed to maintain high annotation quality. The value A is observed by the relationship between the annotation quality calculated by the person in charge or the expert through the A3C algorithm and the number of completed annotation tasks;

[0143] For general labelers, every time they start a new labeling task, there is an initial warm-up period until they are fully focused on the task. Then, over time, the labelers' attention will fluctuate. Therefore, making general labelers focus on a specific field or task type will make the labeling results they provide more reliable. When their attention decreases, it is wise to stop assigning additional tasks. Appropriate incentive strategies can also be used to obtain more reliable labeling results.

[0144] The dynamic task adjustment of general labelers is divided into the following steps:

[0145] Step S3.1: Track the changes in the annotation quality of the annotators during the annotation process through the A3C algorithm, so as to dynamically capture the fluctuations in the attention level, wherein the annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of general annotators;

[0146] Step S3.2: If the annotation quality of the general annotator is lower than B as evaluated in step S3.1, the attention is reduced, and the task assignment is reduced or rest time is given in time to help the annotator to adjust themselves to prevent the decrease of attention caused by long-term continuous work or lack of proper rest, or to improve attention through incentives. The B value is observed by the relationship between the annotation quality calculated by the person in charge or the expert through the A3C algorithm and the number of completed annotation tasks;

[0147] Step S3.3: When the annotation quality of general annotators is improved to higher than C through step S3.2, attention is increased, and the amount of tasks or the difficulty of tasks is increased, that is, specific fields or task types are expanded to ensure that the amount of tasks for each person is appropriate and the overall annotation efficiency is not reduced. The C value is observed by the relationship between the annotation quality calculated by the A3C algorithm by the person in charge or the expert and the number of completed annotation tasks;

[0148] The annotation speed in step 3.1 or step S3.1 refers to the amount of annotation tasks completed per unit time, which is used to measure the efficiency of the annotator or system. The specific calculation method varies according to different annotation tasks. The following is a general calculation method for annotation speed:

[0149]

[0150] Completed annotations: The number of annotation tasks completed by the annotator within the specified time.

[0151] Total time: The total time it takes for annotators to complete these tasks, usually measured in hours, minutes, etc.

[0152] The accuracy in step 3.1 or step S3.1 is:

[0153]

[0154] The incentives described in step S3.2 include points or rewards;

[0155] The points specifically include: task completion points and high-quality annotation points;

[0156] The rewards specifically include: money, honor, and gifts.

[0157] The A3C algorithm (Asynchronous Advantage Actor-Critic) described in the above steps is a policy gradient-based reinforcement learning method that achieves fast and stable learning results through asynchronous training of multiple agents.

[0158] The training process of the A3C algorithm is divided into the following steps:

[0159] (1) Initialize neural network parameters;

[0160] (2) Create multiple parallel training threads, each of which independently runs an agent to interact with the environment and uses the Actor and Critic network to approximate the strategy and value;

[0161] (3) Each thread selects an action based on the current policy network and observes the new state and reward, storing this information in the experience replay buffer;

[0162] (4) When a thread reaches a certain number of time steps or the trajectory ends, the thread samples the data in the experience replay buffer and performs gradient updates by calculating the advantage function;

[0163] (5) After each thread performs a certain number of gradient updates, it passes the updated parameters to the main thread for overall parameter update;

[0164] (6) Repeat the above steps until the predetermined training rounds are reached or the termination condition is met.

Claims

1. A method for quality control of labeled data in a multi-person collaborative scenario, characterized in that: The following steps are involved: Step 1: First, select samples from the dataset to be labeled, and then the labelers label the selected samples to obtain the sample labeling results. Referring to the sample labeling dataset, calculate the accuracy, recall rate, and F1 score of each labeler's labeling results; Step 2: Based on the accuracy, recall, and F1 score obtained in step 1, the labelers are divided into senior labelers and general labelers. Step 3: Dynamically assign tasks to the senior labelers and general labelers divided in step 2.

2. According to the method for controlling the quality of annotation data in a multi-person collaboration scenario according to claim 1, it is characterized in that: The step 1 specifically includes: First, according to the pre-established annotation specifications and standard operating procedures, samples are selected from the dataset to be annotated for annotation, and a sample annotation dataset is produced; The examples cover all label types and include annotation tasks of different difficulty levels to ensure the diversity of the sample data; The label type is customized by the user according to different labeling tasks, taking the text labeling task of named entity recognition as an example; The simple difficulty of the text annotation task of named entity recognition is to annotate a category of named entities in the text; A medium-difficulty text annotation task for named entity recognition: annotating multiple categories of named entities; Complex and difficult text annotation tasks for named entity recognition: annotating entities in long texts and solving the coreference resolution problem; Next, all the labelers are trained to understand the labeling requirements, labeling methods and the use of labeling tools. Then the sample data is distributed to all the labelers and labeling instructions are provided. The labelers independently complete the labeling of the sample data and submit the results within the specified time. The annotation instructions provide annotation specifications and operation procedures; Finally, referring to the sample annotation dataset, we calculated the accuracy, recall, and F1 score of each annotator’s annotation results to assess whether the annotator has correctly understood and followed the annotation instructions: Confusion Matrix in Classification Problems In the matrix, 1 represents the positive class and 0 represents the negative class; TP (True Positives): When the actual result and the predicted result are both 1, it is a true class, indicating the number of positive classes correctly classified as positive classes; TN (True Negative): When the actual result and the predicted result are both 0, it is a true negative class, indicating the number of negative classes that are correctly classified as negative classes; FP (False Positive): When the actual result is 0 but the predicted result is 1, it is a false positive, indicating the number of negative classes that are mistakenly classified as positive classes; FN (False Negative): When the actual result is 1 but the predicted result is 0, it is a false negative class, indicating the number of positive classes that are mistakenly classified as negative classes; The accuracy is calculated as follows: Accuracy is the ratio of correctly classified samples to the total number of samples, indicating the overall predictive ability of the model. The calculation formula is: The calculation method of recall rate is: Recall, also known as recall rate, is the probability of being correctly predicted as a positive sample among samples that are actually positive. The calculation formula is: F1 score calculation method: The F1 score is the harmonic mean of precision and recall, which combines the performance of the two. It strikes a balance between precision and recall and is suitable for situations where there is an imbalance between classes in classification tasks: Among them, precision is the proportion of samples that are actually positive among all samples predicted to be positive. This is an indicator to measure the accuracy of the results, especially for cases where positive prediction is concerned. The calculation formula is:

3. According to the method for controlling the quality of annotation data in a multi-person collaboration scenario as described in claim 1, it is characterized in that: The step 2 specifically includes: The senior labelers are those with an accuracy rate higher than m% and an error rate lower than (100-m)%; The general labelers are labelers whose accuracy is within the range of (n-m)% and whose error rate is within the range of (100-m)-(100-n)%; Where 0≤n <m≤100。 4. The method for controlling the quality of annotation data in a multi-person collaboration scenario according to claim 1, characterized in that: The step 3 specifically includes: The dynamic task adjustment of senior annotators in the actual annotation process is divided into the following steps: Step 3.1: Use the A3C algorithm to track changes in the annotation quality of senior annotators during the annotation process, so as to dynamically capture fluctuations in attention levels. The annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of senior annotators; Step 3.2: If the annotation quality of the senior annotator is lower than A as evaluated in step 3.1, the attention is reduced, and the type or content of the annotation task is changed to maintain high annotation quality. The value A is observed by the relationship between the annotation quality calculated by the person in charge or the expert through the A3C algorithm and the number of completed annotation tasks; Generally, the dynamic task adjustment of labelers is divided into the following steps: Step S3.1: Track the changes in the annotation quality of general annotators during the annotation process through the A3C algorithm, so as to dynamically capture the fluctuation of the attention level, wherein the annotation quality includes: the annotation speed and accuracy obtained by real-time monitoring of the general annotators; Step S3.2: If the annotation quality of the general annotator is lower than B as evaluated in step S3.1, the attention is reduced, and the task assignment is reduced in time or rest time is given, or incentives are used to improve attention. The B value is observed by the relationship between the annotation quality calculated by the A3C algorithm by the person in charge or the expert and the number of completed annotation tasks; Step S3.3: When the annotation quality of general annotators is improved to higher than C through step S3.2, attention is increased, and the amount of tasks or task difficulty is increased, that is, specific fields or task types are expanded. The C value is observed by the relationship between the annotation quality calculated by the A3C algorithm by the person in charge or the expert and the number of completed annotation tasks; The labeling speed in step 3.1 or step S3.1 refers to the amount of labeling tasks completed per unit time, which is used to measure the efficiency of the labeler or system. The specific calculation method varies according to different labeling tasks. The following is a general calculation method for labeling speed: Completed annotation quantity: the number of annotation tasks completed by the annotator within the specified time; Total time: the total time it takes for the annotator to complete these tasks, measured in hours, minutes, etc. The accuracy in step 3.1 or step S3.1 is: Confusion matrix in classification problems: In the matrix, 1 represents the positive class and 0 represents the negative class; TP (True Positives): When the actual result and the predicted result are both 1, it is a true class, indicating the number of positive classes correctly classified as positive classes; TN (True Negative): When the actual result and the predicted result are both 0, it is a true negative class, indicating the number of negative classes that are correctly classified as negative classes; FP (False Positive): When the actual result is 0 but the predicted result is 1, it is a false positive, indicating the number of negative classes that are mistakenly classified as positive classes; FN (False Negative): When the actual result is 1 but the predicted result is 0, it is a false negative class, indicating the number of positive classes that are mistakenly classified as negative classes.

5. The method for controlling the quality of labeled data in a multi-person collaborative scenario according to claim 4, characterized in that: The incentives described in step S3.2 include points or rewards; The points specifically include: task completion points and high-quality annotation points; The rewards specifically include: money, honor, gifts; The A3C algorithm (Asynchronous Advantage Actor-Critic) described in the above steps is a policy gradient-based reinforcement learning method that achieves fast and stable learning results through asynchronous training of multiple agents.

6. The method for controlling the quality of labeled data in a multi-person collaboration scenario according to claim 4, characterized in that: The training process of the A3C algorithm is divided into the following steps: (1) Initialize neural network parameters; (2) Create multiple parallel training threads, each of which independently runs an agent to interact with the environment. And use Actor and Critic networks to achieve approximation of strategies and values; (3) Each thread selects an action based on the current policy network and observes the new state and reward, storing this information in the experience replay buffer; (4) When a thread reaches a certain number of time steps or the trajectory ends, the thread samples the data in the experience replay buffer and performs gradient updates by calculating the advantage function; (5) After each thread performs a certain number of gradient updates, it passes the updated parameters to the main thread for overall parameter update; (6) Repeat the above steps until the predetermined training rounds are reached or the termination condition is met.

Images