A method, device and medium for improving quality of data labeling crowdsourcing

Abstract

The application discloses a data labeling crowdsourcing quality improvement method, device and medium, the method comprises the following steps: determining the task demand of a data labeling task, determining the corresponding labeling personnel according to the task demand, issuing the data labeling task to the labeling personnel, so that the labeling personnel executes the data labeling task; collecting the labeling result of the data labeling task, determining the labeling result quantity according to the labeling result, and performing consistency verification on the labeling result quantity according to the pre-set verification standard; if the consistency verification is qualified, the labeling result is saved, and the labeling result is marked; if the consistency verification is not qualified, the data labeling task is reissued for execution until the consistency verification of the labeling result quantity is qualified. The process logic of the application is clear, the personnel is accurately matched from the task demand, the result is strictly verified, and the task is reissued if it is unqualified, so that the accuracy and high-quality output of the data labeling result are guaranteed.

Description

Technical Field

[0001] This application relates to the field of big data governance technology, and in particular to a method, device and medium for improving the quality of data annotation crowdsourcing. Background Technology

[0002] Data annotation crowdsourcing, by leveraging distributed human resources to improve data annotation efficiency, is an important model for driving the development of artificial intelligence. Data annotation is essential in several ways. First, it improves the quality of data supply. Data annotation is a fundamental step in AI training and directly affects model performance. OpenAI invested thousands of people and hundreds of millions of dollars in data annotation when training its GPT series models; high-quality corpora are a key factor in their model's leading performance. Currently, the market generally suffers from low data quality and low utilization rates. Crowdsourcing can quickly integrate scattered data resources to compensate for these deficiencies. Second, it accelerates the realization of data value. The widespread use of crowdsourcing improves data annotation efficiency. Taking medical image annotation as an example, this model can be used to train accurate disease diagnosis models, thereby realizing the transformation of data value. Third, it promotes employment and industrial upgrading. The data annotation industry is gradually developing towards intelligence and specialization. Crowdsourcing can absorb a large amount of labor, helping to alleviate employment pressure. For example, image annotation accounts for 40% to 70% of the development cycle of computer vision (CV) projects, and distributed annotation can reduce resource consumption.

[0003] Crowdsourcing of data annotation offers numerous advantages. In terms of cost-effectiveness, traditional data annotation relies on professional teams or companies, resulting in high costs and difficulty in meeting large-scale demands. The crowdsourcing model significantly reduces annotation costs by integrating a large number of part-time personnel or small teams, making it particularly suitable for simple tasks such as bounding box annotation and point-to-point annotation. Regarding rapid response, the crowdsourcing model can quickly aggregate massive amounts of annotation resources, making it particularly suitable for urgent projects such as smart cities and security monitoring. For example, a smart city video annotation project used a crowdsourcing model to complete the tracking of 20,000 vehicles and the annotation of over 1,000 event types in a short period. In terms of flexibility, the crowdsourcing model supports flexible task allocation and quality control mechanisms. Platforms can implement reassessment processes and task tiering systems, allowing high-accuracy annotators to unlock more complex tasks.

[0004] Currently, data annotation crowdsourcing faces several pressing issues. Firstly, the workload is excessively high. Most data annotation tasks are simple and repetitive, easily leading to fatigue for annotators and negatively impacting annotation quality. Secondly, quality control is challenging. Annotation quality is affected by various factors, such as differences in annotators' skill levels, misunderstandings of annotation requirements, some annotators rushing through tasks for quick payment, and even organized cheating. These factors result in a single delivery success rate of less than 50%, and even after three deliveries, the success rate remains below 90%, highlighting the difficulty in quality control in data crowdsourcing. Summary of the Invention

[0005] To address the aforementioned issues, this application proposes a method for improving the quality of data annotation crowdsourcing, comprising: determining the task requirements of a data annotation task; identifying corresponding annotators based on the task requirements; issuing the data annotation task to the annotators so that they can execute the data annotation task; collecting the annotation results of the data annotation task; determining the number of annotation results based on the annotation results; and performing a consistency check on the number of annotation results according to a pre-set verification standard; if the consistency check is successful, saving the annotation results and marking them; if the consistency check fails, reissuing the data annotation task for execution until the consistency check of the number of annotation results is successful.

[0006] In one example, the consistency of the number of annotation results is checked according to a pre-set verification standard. Specifically, this includes: determining all data annotation results, determining consistent data annotation results based on all annotation results, determining the proportion of consistent data annotation results, comparing the proportion of consistent data annotation results with a pre-set qualified proportion threshold; if the proportion of consistent data annotation results is greater than the qualified proportion threshold, the consistent data annotation results are determined to be qualified, and the consistent data annotation results are saved as qualified annotation results.

[0007] In one example, after the data annotation task is redistributed and executed, the method further includes: determining a pre-set redistribution number threshold; when the number of redistributions of the data annotation task reaches the redistribution number threshold, determining whether the consistency check of the number of annotation results is qualified; if the consistency check is not qualified, the task publisher reviews the data annotation task to determine the data annotation results and applies a pre-set special mark to the data annotation results.

[0008] In one example, the method further includes: sampling completed data annotation tasks according to a pre-set sampling ratio to determine the selected data annotation tasks; reviewing the selected data annotation tasks to determine whether there is any collusion in the selected data annotation tasks; if there is collusion in the selected data annotation tasks, marking the annotation results of the corresponding data annotation tasks and re-executing the data annotation tasks.

[0009] In one example, the method further includes: determining the correct answer rate of the completed data annotation task; evaluating the correct answer rate according to a pre-set scoring system to determine the credit score of the data annotation task; determining the marker of the completed data annotation task; determining whether there is any collusion in the data annotation task based on the marker; and deducting the credit score according to a pre-set scoring standard if collusion is found.

[0010] In one example, the method further includes: determining the unit price corresponding to the data annotation task, and determining the annotation quality and credit rating of the annotator corresponding to the data annotation task; and determining the bonus for the data annotation task based on the unit price, the annotation quality, and the credit rating.

[0011] In one example, determining the corresponding labeling personnel based on the task requirements specifically includes: determining the corresponding key data items based on the task requirements, and then determining the labeling personnel assigned to the data labeling task based on the key data items. The key data items include personnel ID, professional field, credit rating, and the region to which the login IP address belongs.

[0012] In one example, the method further includes: determining the projects of the data annotation task, the projects including annotation personnel management, project configuration, data annotation, result inspection, credit evaluation, and bonus calculation; determining the project configuration of the data annotation task, the project configuration including professional field, credit level, number of people with duplicate annotations, minimum number of duplicate annotations, minimum number of qualified annotations, and sampling rate of annotation results.

[0013] On the other hand, this application also proposes a quality improvement device for data annotation crowdsourcing, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the data annotation crowdsourcing quality improvement device to perform: the method described in any of the examples above.

[0014] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to be the method described in any of the examples above.

[0015] This application precisely matches annotators to task requirements, considering factors such as personnel ID and professional field to ensure personnel suitability. It rigorously verifies the consistency of the number of annotation results, setting a threshold for the qualified percentage to ensure result quality. A threshold for the number of reissues is set; exceeding this threshold requires review by the publisher to prevent infinite loops and ensure efficiency. A random inspection mechanism is also included to promptly detect collusion and cheating, and to re-execute the task, maintaining fairness and impartiality. Furthermore, a credit score is awarded based on the correct answer rate, with points deducted for cheating, strengthening quality constraints. In addition, bonuses are determined by combining unit price, annotation quality, and credit rating to incentivize annotators to improve quality. Finally, the project and configuration details, such as professional field and random inspection ratio, are clearly defined, making the entire process standardized, orderly, and highly operable. This method comprehensively ensures data annotation quality, forming a complete closed loop from personnel selection to result review, credit evaluation, and bonus incentives, effectively improving the overall quality and efficiency of data annotation crowdsourcing. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a method for improving the quality of data annotation crowdsourcing in an embodiment of this application; Figure 2 This is a schematic diagram of a data annotation crowdsourcing quality improvement device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, in order to solve the above problems, this application provides a method for improving the quality of data annotation crowdsourcing, the method including: S101. Determine the task requirements for the data annotation task, determine the corresponding annotation personnel based on the task requirements, and issue the data annotation task to the annotation personnel so that the annotation personnel can perform the data annotation task.

[0020] In one embodiment, the management of data annotation projects should follow a project management process system to systematically control each data annotation task. Specifically, data annotation projects encompass six core steps: annotator management, project configuration, data annotation, result checking, credit evaluation, and bonus calculation. The project configuration stage requires clearly defining key parameters such as the professional field, credit rating requirements, number of people required to annotate repeatedly, minimum number of repeated annotations, minimum number of qualified annotations, and the sampling rate for annotation results. Each data annotation project contains a batch of data to be annotated, ranging in size from dozens to hundreds of thousands of records. During project execution, multiple roles are involved, including annotators, task issuers, sampling personnel, and project managers. All parties must collaborate to ensure the project's successful completion.

[0021] In one embodiment, the management of data labelers encompasses four key data items: personnel ID, professional field, credit rating, and the region of their login IP address. The personnel ID serves as a unique identifier to accurately distinguish each labeler; the professional field information is used to select labelers with relevant professional backgrounds based on the specific needs of the data labeling task, ensuring the professionalism and accuracy of the labeling results; the credit rating, as an important indicator of labeler reliability, helps to select labelers with good credit in their professional fields, further improving labeling quality; and the region of the login IP address is used to identify the geographical distribution of labelers. By assigning data labeling tasks to personnel in different regions, it effectively avoids labelers in concentrated areas producing uniform labeling results and circumventing evaluation rules, thereby ensuring the overall quality of data labeling.

[0022] In one embodiment, for each data annotation task, N annotators with sufficient credit ratings G and from different geographical locations are selected from a suitable professional field P, based on the specific requirements of the task, and data annotation tasks are assigned to them. Here, the professional field P, credit rating G, and number of annotators N are all configurable parameters that can be flexibly set according to the actual needs of each data annotation operation.

[0023] S102. Collect the annotation results of the data annotation task, determine the number of annotation results based on the annotation results, and perform consistency verification on the number of annotation results according to the pre-set verification standard.

[0024] After completing the execution of the data annotation task, collect all data annotation results, and record in detail the identity information of each data annotator and key data such as the annotation time. Subsequently, for each data annotation task, it is necessary to calculate whether the number of collected data annotation results reaches the preset minimum requirement. Among them, the minimum number M of data annotation results is a parameter that can be configured according to the actual needs of each data annotation service, and its set value must strictly meet the condition of being greater than zero and less than or equal to the number N of data annotation tasks issued, that is, ensure 0 < M ≤ N.

[0025] S103. If the consistency check is qualified, save the annotation result and mark the annotation result.

[0026] After the number of collected data annotation results reaches the preset minimum requirement M, it is necessary to further calculate the consistency ratio of these results. Specifically, if the ratio of the number of consistent data annotation results to the total collected number exceeds the preset qualified ratio threshold C / M, where the minimum qualified number C is a configurable parameter, set according to the needs of each data annotation service and satisfying 0 < C ≤ M, then it is determined that the quality of the data annotation results is qualified, and these consistent results are saved as correctly annotated data.

[0027] S104. If the consistency check is not qualified, reissue the data annotation task for execution until the consistency check of the annotation result quantity is qualified.

[0028] Conversely, if the consistency ratio is lower than C / M, it indicates that the quality of the data annotation results does not meet the standard. At this time, the above data annotation steps need to be re-executed, that is, re-screen the annotators and issue the task until the result quality is qualified or the number of times the data annotation task is reissued reaches the preset threshold.

[0029] In one embodiment, when the number of times the data annotation task is reissued reaches the preset threshold, the task publisher will be involved in handling. Specifically, the task publisher needs to carefully review the data annotation task and personally give the final data annotation result. This result will be saved as a data annotation result with qualified quality and will be specially marked to clearly distinguish it from the results submitted by ordinary annotators.

[0030] In one embodiment, after the data annotation project is completed, it is necessary to randomly select the data annotation results that have been determined by the system to be of qualified quality according to the preset sampling ratio R. This ratio is a configurable parameter and can be adjusted according to the specific needs of each data annotation service. During the review process, the sampling personnel will carefully review these annotation results, focusing on checking for organized joint cheating behavior.

[0031] If obvious defects are found in the selected annotation results, it is preliminarily determined that there is a suspicion of organized collusion and cheating. At this time, all qualified annotation results related to the annotation task should be uniformly marked as "suspected of cheating", and the task should be re-annotated in strict accordance with the established annotation process.

[0032] In one embodiment, after a data annotation project is completed, each data annotator's credit score is calculated based on the actual annotation quality. If an annotator is suspected of cheating, 10 credit points are deducted. If an annotator's correct answer rate reaches or exceeds 95%, 1 credit point is awarded. If the correct answer rate is between 90% and 95%, 0.8 credit points are awarded; if the correct answer rate is between 80% and 90%, 0.6 credit points are awarded; if the correct answer rate is between 70% and 80%, 0.4 credit points are awarded; if the correct answer rate is between 60% and 70%, it is determined that there is a lack of knowledge or skills, and no credit points are awarded; if the correct answer rate is below 60%, it is determined that there is a bad attitude, and 0.5 credit points are deducted as punishment.

[0033] In one embodiment, upon completion of a data annotation project, a bonus is calculated and awarded based on a comprehensive assessment of the annotation task's difficulty, the annotation quality of each data annotator, and their credit rating. Difficulty is represented by a unit price, while annotation quality is specifically reflected in the number of correct answers. The unit price for the data annotation task is a configurable parameter, determined by the project manager based on the task's actual difficulty, quality requirements, and the final outcome of commercial negotiations.

[0034] like Figure 2 As shown in the illustration, this application also provides a quality improvement device for data annotation crowdsourcing, comprising: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable a data labeling crowdsourcing quality improvement device to perform the method as described in any of the embodiments above.

[0035] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which are configured as described in any of the above embodiments.

[0036] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0037] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0038] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0039] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0040] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0041] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0042] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0045] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0046] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0047] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0048] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0049] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for improving the quality of data annotation crowdsourcing, characterized in that, include: Determine the task requirements for the data annotation task, identify the corresponding annotators based on the task requirements, and issue the data annotation task to the annotators so that the annotators can perform the data annotation task; Collect the annotation results of the data annotation task, determine the number of annotation results based on the annotation results, and perform consistency verification on the number of annotation results according to the pre-set verification criteria; If the consistency check passes, the annotation results are saved and marked. If the consistency check fails, the data annotation task will be reissued and executed until the consistency check of the number of annotation results passes.

2. The method according to claim 1, characterized in that, The consistency of the labeled results is verified according to pre-set verification criteria, specifically including: Determine all data annotation results, determine consistent data annotation results based on all annotation results, determine the proportion of consistent data annotation results, and compare the proportion of consistent data annotation results with a pre-set qualified proportion threshold. If the quantity ratio is greater than the qualified ratio threshold, the consistent data annotation result is determined to be qualified, and the consistent data annotation result is saved as a qualified annotation result.

3. The method according to claim 1, characterized in that, After the data annotation task is reissued and executed, the method further includes: A pre-set threshold for the number of re-distributions is determined. When the number of re-distributions of the data annotation task reaches the threshold, it is determined whether the consistency check of the number of annotation results is qualified. If the consistency check fails, the task publisher will review the data annotation task to determine the data annotation results and apply a pre-set special mark to the data annotation results.

4. The method according to claim 1, characterized in that, The method further includes: Completed data labeling tasks are randomly selected based on a pre-set sampling ratio to determine the selected data labeling tasks. The selected data labeling tasks are then reviewed to determine whether there is any collusion or cheating involved. If collusion or cheating is found in the selected data annotation task, the annotation results of the corresponding data annotation task will be marked, and the data annotation task will be re-executed.

5. The method according to claim 1, characterized in that, The method further includes: The correct answer rate of the completed data annotation task is determined, and the correct answer rate is evaluated according to a pre-set scoring system to determine the credit score of the data annotation task. The completed data annotation task is marked, and based on the mark, it is determined whether there is any collusion in the data annotation task. If collusion is found, the credit score is deducted according to the pre-set scoring criteria.

6. The method according to claim 1, characterized in that, The method further includes: Determine the unit price corresponding to the data annotation task, and determine the annotation quality and credit rating of the annotator corresponding to the data annotation task; The bonus for the data labeling task is determined based on the unit price, the labeling quality, and the credit rating.

7. The method according to claim 1, characterized in that, The corresponding labeling personnel are determined based on the task requirements, specifically including: Based on the task requirements, the corresponding key data items are determined, and the annotation personnel assigned to the data annotation task are determined based on the key data items. The key data items include personnel ID, professional field, credit rating, and the region to which the login IP address belongs.

8. The method according to claim 1, characterized in that, The method further includes: The data annotation task is defined as including the following components: annotation personnel management, project configuration, data annotation, result checking, credit evaluation, and bonus calculation. The project configuration for the data annotation task is determined, including the professional field, credit rating, number of people with duplicate annotations, minimum number of duplicate annotations, minimum number of qualified annotations, and sampling rate of annotation results.

9. A quality improvement device for data annotation crowdsourcing, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the data annotation crowdsourcing quality improvement device to perform the method as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to be the method as described in any one of claims 1-8.

Images