Data annotation method and device, equipment and storage medium
By employing a multi-person annotation and multi-level review method, highly accurate data is selected for training machine learning models. This solves the problem of data annotation uncertainty in scenarios such as lung function tests, and improves data quality and training efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SAIKE XIAMEN MEDICAL DEVICES CO LTD
- Filing Date
- 2024-11-22
- Publication Date
- 2026-05-22
AI Technical Summary
In data labeling scenarios lacking clear quantitative standards, especially in fields such as lung function tests, the standards for labeling data are subject to subjective judgment and uncertainty, leading to low data quality and efficiency.
By employing a multi-person annotation method, the probability of correct answers from annotators with adjacent numbers is calculated to filter out data with high accuracy. This data undergoes multi-level review before being used to train a machine learning model.
It improved the quality and efficiency of labeled data, reduced the time cost of model training, and increased the efficiency of model training.
Smart Images
Figure CN122072847A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data annotation technology, and more specifically, to a data annotation method, apparatus, device, and storage medium. Background Technology
[0002] For data annotation in certain specific scenarios, due to the lack of clear quantitative annotation standards, it relies on subjective human judgment. For example, in pulmonary function tests, the quality control judgment mainly relies on the descriptions in guidelines. However, the guidelines do not form complete quantitative standards and are more based on the clinical experience of clinicians. In such scenarios, the standard for annotating data is subject to subjective judgment. For example, different clinicians may have different judgments about whether there is initial hesitation in a pulmonary function curve. Therefore, there is a lot of uncertainty in the data annotation work in such scenarios. This invention mainly solves the problem of data screening and review in such scenarios, improving the quality and efficiency of annotated data. Summary of the Invention
[0003] The purpose of this invention is to provide a data annotation method, apparatus, device, and storage medium to improve the above-mentioned problems.
[0004] To achieve the above objectives, the embodiments of this application provide the following technical solutions:
[0005] On the one hand, embodiments of this application provide a data annotation method, the method comprising:
[0006] Obtain multiple data points to be labeled, the ID of each labeler, and the labeling information of each labeler for each data point to be labeled. The labeler's name must be at least 3 characters long.
[0007] Based on the annotation information corresponding to each labeled data, the probability of correct answering for two adjacent annotators is calculated using a preset formula;
[0008] The labeled data is initially filtered based on the total correct answer probability. The labeled data obtained from the initial filtering is then filtered again to obtain the remaining labeled data. The machine learning model is trained based on the remaining labeled data to obtain the labeling model, which is used to label new data to be labeled.
[0009] Optionally, based on the annotation information corresponding to each labeled data point, the probability of correct answers for two adjacent annotators is calculated using a preset formula, including:
[0010] Let A be the probability that the first annotator marks correctly and 1-A be the probability that it marks incorrectly. Let B be the probability that the second annotator marks correctly and 1-B be the probability that it marks incorrectly. The first annotator and the second annotator are two annotators with adjacent numbers.
[0011] For each piece of data to be labeled, analyze whether the labeling information of the first labeler and the second labeler is consistent. If they are consistent, add this piece of data to the dataset. After analyzing each piece of data, count the number of pieces of data to be labeled in the dataset and record it as N; count the number of pieces of data to be labeled that each labeler needs to complete and record it as M.
[0012] Assuming A = B, we can calculate two values of A using formulas (1) and (2), which are:
[0013] AB + (1-A)*(1-B)=N / M (1)
[0014] (1-A)*B + A*(1-B) =1- N / M (2)
[0015] The larger of the two values of A is selected as the probability of the two adjacent annotators answering correctly.
[0016] Optionally, the labeled data can be initially filtered based on the overall correct answer probability. The initially filtered labeled data can then be further filtered to obtain the remaining labeled data, including:
[0017] The system selects the answers with the highest probability of being correct and then filters out the labeled data completed by the two annotators corresponding to each answer, thus completing the initial screening.
[0018] The initially screened labeled data is sent to reviewers at all levels so that they can conduct multi-level reviews of the labeled information. In each level of review, labeled data that is inconsistent with the review results is deleted, and finally the remaining labeled data is obtained.
[0019] Optionally, the machine learning model can be trained based on the remaining labeled data to obtain a labeled model, including:
[0020] A random grouping algorithm is used to select training and test sets from the remaining labeled data to train the machine learning model; the data to be labeled is used as input, and the labeling information corresponding to the data to be labeled is used as output.
[0021] Secondly, embodiments of this application provide a data annotation apparatus, the apparatus comprising:
[0022] The acquisition module is used to acquire multiple data to be labeled, the ID of each labeler, and the labeling information of each labeler for each data to be labeled. The labeler's name must be at least 3 characters long.
[0023] The calculation module is used to calculate the probability of correct answers for two adjacent annotators based on the annotation information corresponding to each annotated data using a preset formula;
[0024] The annotation module is used to initially filter the labeled data based on the total correct answer probability, further filter the initially filtered labeled data to obtain the remaining labeled data, train the machine learning model based on the remaining labeled data to obtain the annotation model, and the annotation model is used to annotate new data to be labeled.
[0025] Optional, the computation module includes:
[0026] The marking unit is used to record the probability of the first annotator marking correctly as A, and the probability of incorrect marking as 1-A; the probability of the second annotator marking correctly as B, and the probability of incorrect marking as 1-B; the first annotator and the second annotator are two annotators with adjacent numbers.
[0027] The statistical unit is used to analyze whether the annotation information of the first annotator and the second annotator is consistent for each data to be labeled. If they are consistent, the data to be labeled is added to the dataset. After the analysis of each data to be labeled is completed, the number of data to be labeled in the statistical dataset is counted as N; the number of data to be labeled that each annotator needs to complete is counted as M.
[0028] The calculation unit is used to calculate two values of A based on formulas (1) and (2), assuming A = B, and the formulas (1) and (2) are:
[0029] AB + (1-A)*(1-B)=N / M (1)
[0030] (1-A)*B + A*(1-B) =1- N / M (2)
[0031] The selection unit is used to choose the larger of the two values of A as the probability of correct answer for the two adjacent annotators.
[0032] Optional annotation modules include:
[0033] The filtering unit is used to filter out the data with the highest probability of answering correctly and to filter out the labeled data completed by the two annotators corresponding to it, thus completing the initial filtering.
[0034] The review unit is used to send the initially screened labeled data to reviewers at all levels so that they can conduct multi-level reviews of the labeled information. In each level of review, labeled data that is inconsistent with the review results is deleted, and finally the remaining labeled data is obtained.
[0035] Optional annotation modules include:
[0036] The training unit is used to select training and test sets from the remaining labeled data using a random grouping algorithm to train the machine learning model; the data to be labeled is taken as input, and the labeling information corresponding to the data to be labeled is taken as output.
[0037] Thirdly, embodiments of this application provide a data annotation device, which includes a memory and a processor. The memory stores a computer program; the processor executes the computer program to implement the steps of the data annotation method described above.
[0038] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data annotation method.
[0039] The beneficial effects of this invention are as follows:
[0040] The invention provides an optimized solution for selecting labeled data. By selecting data with a relatively high accuracy probability for machine learning training, the quality of the training dataset can be improved, thereby increasing the efficiency of model training and reducing the time cost of model parameter tuning.
[0041] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a schematic diagram of the data annotation method described in the embodiments of the present invention;
[0044] Figure 2 This is a schematic diagram of the data annotation device structure described in the embodiments of the present invention;
[0045] Figure 3 This is a schematic diagram of the data annotation device structure described in an embodiment of the present invention. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0047] It should be noted that similar reference numerals or letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0048] Example 1
[0049] like Figure 1 As shown, this embodiment provides a data annotation method, which includes steps S1, S2 and S3.
[0050] Step S1: Obtain multiple data to be labeled, the number of each labeler, and the labeling information of each labeler for each data to be labeled. The number of labelers must be at least 3.
[0051] In this step, each annotator can be numbered, for example, 1, 2, 3, 4...; if there are 200 data points to be annotated, then each annotator must annotate all 200 data points; the data to be annotated and the corresponding annotation information can be:
[0052] Key images from each lung function test, such as flow-volume curves and volume-time curves, as well as lung function index values that assist in quality control and result interpretation; referring to the latest lung function guidelines for the description of test quality control and result interpretation standards, and combining the obtained test images and index values, the quality control artifacts, curve acceptability and usability, overall test quality level, lung ventilation function type, and ventilation function severity of each test are labeled to obtain the labeling information for each case;
[0053] Step S2: Based on the annotation information corresponding to each labeled data, calculate the probability of correct answering for two adjacent annotators using a preset formula;
[0054] The specific implementation steps of this step include steps S21-S24;
[0055] Step S21: Let A be the probability that the first annotator marks correctly and 1-A be the probability that it marks incorrectly; let B be the probability that the second annotator marks correctly and 1-B be the probability that it marks incorrectly; the first annotator and the second annotator are two annotators with adjacent numbers.
[0056] Step S22: For each piece of data to be labeled, analyze whether the labeling information of the first labeler and the second labeler is consistent. If they are consistent, put this piece of data to be labeled into the dataset. After each piece of data to be labeled is analyzed, count the number of pieces of data to be labeled in the dataset and record it as N; count the number of pieces of data to be labeled that each labeler needs to complete and record it as M.
[0057] Step S23: Assuming A = B, calculate two values of A according to formula (1) and formula (2). Formula (1) and formula (2) are as follows:
[0058] AB + (1-A)*(1-B)=N / M (1)
[0059] (1-A)*B + A*(1-B) =1- N / M (2)
[0060] Step S24: Select the larger value from the two values of A as the probability of the two adjacent annotators answering correctly.
[0061] Steps S21 to S24 can be understood as follows: Assuming AB labels 100 identical data points, and 90 of them have identical labeling information, then:
[0062] P(1|1)+P(0|0)=90 / 100;
[0063] P(0|1)+P(1|0)=10 / 100;
[0064] AB + (1-A)*(1-B) = 0.9
[0065] (1-A)*B+A*(1-B)=0.1
[0066] Simplify
[0067] AB+1-A-B+AB=0.9
[0068] Assume that A and B have the same probability of answering correctly.
[0069] -2A + 2A*A + 0.1 = 0;
[0070] Solving for A, we get A = B = 0.947 or 0.053. In this embodiment, all the annotators have received annotation training, so the probability of correct annotation is assumed to be much greater than the probability of incorrect annotation. Furthermore, since the probability of correct annotation is greater than the probability of incorrect annotation, the probability of A and B being correct is 0.947.
[0071] Step S3: Perform preliminary screening of the labeled data based on the total correct answer probability. Then, further screen the labeled data obtained from the preliminary screening to obtain the remaining labeled data. Train the machine learning model based on the remaining labeled data to obtain the labeling model. The labeling model is used to label new data to be labeled.
[0072] In this step, the labeled data can be understood as follows: if the data to be labeled is H and the corresponding labeling information is G, then the labeled data includes both H and G. The labeled data is initially filtered based on the total correct answer probability. The labeled data obtained from the initial filtering is then filtered again to obtain the remaining labeled data. The specific implementation steps include steps S31 and S32.
[0073] Step S31: Select the answer with the highest probability of being correct and filter out the labeled data completed by the two annotators corresponding to it to complete the initial screening;
[0074] Step S32: Send the pre-screened labeled data to the reviewers at all levels so that the reviewers can conduct multi-level review of the labeled information; in each level of review, the labeled data that is inconsistent with the review results is deleted, and finally the remaining labeled data is obtained.
[0075] In this step, the initially screened labeled data can be sent to reviewers for multiple rounds of review. The reviewers can be the same or different in each round; it is better if they are different. In each round of review, the review result is the labeling information given by the reviewer. If the labeling information given by the reviewer is inconsistent with the labeling information corresponding to this labeled data, then this labeled data is deleted. After multiple rounds of review, the remaining labeled data is obtained.
[0076] In step S3, the machine learning model is trained based on the remaining labeled data to obtain the labeled model. The specific implementation steps include step S33:
[0077] Step S33: Use a random grouping algorithm to select training and test sets from the remaining labeled data to train the machine learning model; wherein, the data to be labeled is used as input and the labeling information corresponding to the data to be labeled is used as output.
[0078] In this embodiment, the remaining labeled data can be used to train the convolutional neural network model to obtain a labeled model.
[0079] This embodiment provides an optimized solution for selecting labeled data. By selecting data with a relatively high accuracy probability for machine learning training, the quality of the training dataset can be improved, thereby increasing the efficiency of model training and reducing the time cost of model parameter tuning.
[0080] Example 2
[0081] like Figure 2 As shown in the figure, this embodiment provides a data annotation device, which includes an acquisition module 1, a calculation module 2 and an annotation module 3.
[0082] Module 1 is used to acquire multiple data to be labeled, the ID of each labeler, and the labeling information of each labeler for each data to be labeled. The labeler's name must be at least 3 characters long.
[0083] Calculation module 2 is used to calculate the probability of correct answers for two adjacent annotators based on the annotation information corresponding to each annotated data using a preset formula;
[0084] The annotation module 3 is used to perform preliminary screening of the labeled data based on the total correct answer probability, further screen the labeled data obtained from the preliminary screening to obtain the remaining labeled data, train the machine learning model based on the remaining labeled data to obtain the annotation model, and the annotation model is used to annotate new data to be annotated.
[0085] In one specific embodiment of this disclosure, the calculation module 2 further includes a marking unit 21, a statistics unit 22, a calculation unit 23, and a selection unit 24.
[0086] The marking unit 21 is used to record the probability of the first marking person marking correctly as A, and the probability of incorrect marking as 1-A; the probability of the second marking person marking correctly as B, and the probability of incorrect marking as 1-B; the first marking person and the second marking person are two marking persons with adjacent numbers.
[0087] The statistical unit 22 is used to analyze whether the annotation information of the first annotator and the second annotator is consistent for each data to be annotated. If they are consistent, the data to be annotated is put into the dataset. After each data to be annotated is analyzed, the number of data to be annotated in the dataset is counted as N; the number of data to be annotated that each annotator needs to complete is counted as M.
[0088] Calculation unit 23 is used to calculate two values of A based on formula (1) and formula (2), assuming A = B.
[0089] AB + (1-A)*(1-B)=N / M (1)
[0090] (1-A)*B + A*(1-B) =1- N / M (2)
[0091] Selecting unit 24 is used to select the larger value among the two values of A as the probability of correct answer for the two adjacent annotators.
[0092] In one specific embodiment of this disclosure, the annotation module 3 further includes a filtering unit 31 and an auditing unit 32.
[0093] The filtering unit 31 is used to filter out the data with the highest probability of answering correctly and to filter out the labeled data completed by the two annotators corresponding to it, thus completing the initial filtering.
[0094] The review unit 32 is used to send the initially screened labeled data to reviewers at all levels so that they can conduct multi-level reviews of the labeled information. In each level of review, the labeled data that is inconsistent with the review results is deleted, and the remaining labeled data is obtained in the end.
[0095] In one specific embodiment of this disclosure, the annotation module 3 further includes a training unit 33.
[0096] Training unit 33 is used to select training and test sets from the remaining labeled data using a random grouping algorithm to train the machine learning model; wherein the data to be labeled is used as input and the labeling information corresponding to the data to be labeled is used as output.
[0097] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.
[0098] Example 3
[0099] Corresponding to the above method embodiments, this disclosure also provides a data annotation device, and the data annotation device described below can be referred to in correspondence with the data annotation method described above.
[0100] Figure 3 This is a block diagram illustrating a data annotation device 300 according to an exemplary embodiment. For example... Figure 3 As shown, the data annotation device 300 may include a processor 301 and a memory 302. The data annotation device 300 may also include one or more of a multimedia component 303, an I / O interface 304, and a communication component 305.
[0101] The processor 301 controls the overall operation of the data annotation device 300 to complete all or part of the steps in the data annotation method described above. The memory 302 stores various types of data to support the operation of the data annotation device 300. This data may include, for example, instructions for any application or method operating on the data annotation device 300, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 302 or transmitted via communication component 305. The audio component also includes at least one speaker for outputting audio signals. I / O interface 304 provides an interface between processor 301 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 305 is used for wired or wireless communication between the data labeling device 300 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of these. Therefore, the corresponding communication component 305 may include a Wi-Fi module, a Bluetooth module, or an NFC module.
[0102] In one exemplary embodiment, the data annotation device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data annotation method described above.
[0103] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the data annotation method described above. For example, the computer-readable storage medium may be the memory 302 including the program instructions, which may be executed by the processor 301 of the data annotation device 300 to complete the data annotation method described above.
[0104] Example 4
[0105] Corresponding to the above method embodiments, this disclosure also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the data annotation method described above.
[0106] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data annotation method described in the above method embodiments.
[0107] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.
[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A data annotation method, characterized in that, include: Obtain multiple data points to be labeled, the ID of each labeler, and the labeling information of each labeler for each data point to be labeled. The labeler's name must be at least 3 characters long. Based on the annotation information corresponding to each labeled data, the probability of correct answering for two adjacent annotators is calculated using a preset formula; The labeled data is initially filtered based on the total correct answer probability. The labeled data obtained from the initial filtering is then filtered again to obtain the remaining labeled data. The machine learning model is trained based on the remaining labeled data to obtain the labeling model, which is used to label new data to be labeled.
2. The data annotation method according to claim 1, characterized in that, Based on the annotation information corresponding to each labeled data point, the probability of correct answers for two adjacent annotators is calculated using a preset formula, including: Let A be the probability that the first annotator marks correctly and 1-A be the probability that it marks incorrectly. Let B be the probability that the second annotator marks correctly and 1-B be the probability that it marks incorrectly. The first annotator and the second annotator are two annotators with adjacent numbers. For each piece of data to be labeled, analyze whether the labeling information of the first labeler and the second labeler is consistent. If they are consistent, add this piece of data to the dataset. After analyzing each piece of data, count the number of pieces of data to be labeled in the dataset and record it as N; count the number of pieces of data to be labeled that each labeler needs to complete and record it as M. Assuming A = B, we can calculate two values of A using formulas (1) and (2), which are: AB + (1-A)*(1-B)=N / M (1) (1-A)*B + A*(1-B) =1- N / M (2) The larger of the two values of A is selected as the probability of the two adjacent annotators answering correctly.
3. The data annotation method according to claim 1, characterized in that, The labeled data is initially filtered based on the overall correct answer probability. This initial filtering is then further filtered to obtain the remaining labeled data, including: The system selects the answers with the highest probability of being correct and then filters out the labeled data completed by the two annotators corresponding to each answer, thus completing the initial screening. The initially screened labeled data is sent to reviewers at all levels so that they can conduct multi-level reviews of the labeled information. In each level of review, labeled data that is inconsistent with the review results is deleted, and finally the remaining labeled data is obtained.
4. The data annotation method according to claim 3, characterized in that, The machine learning model is trained based on the remaining labeled data to obtain a labeled model, including: A random grouping algorithm is used to select training and test sets from the remaining labeled data to train the machine learning model; the data to be labeled is used as input, and the labeling information corresponding to the data to be labeled is used as output.
5. A data annotation device, characterized in that, include: The acquisition module is used to acquire multiple data to be labeled, the ID of each labeler, and the labeling information of each labeler for each data to be labeled. The labeler's name must be at least 3 characters long. The calculation module is used to calculate the probability of correct answers for two adjacent annotators based on the annotation information corresponding to each annotated data using a preset formula; The annotation module is used to initially filter the labeled data based on the total correct answer probability, further filter the initially filtered labeled data to obtain the remaining labeled data, train the machine learning model based on the remaining labeled data to obtain the annotation model, and the annotation model is used to annotate new data to be labeled.
6. The data annotation device according to claim 5, characterized in that, The calculation module includes: The marking unit is used to record the probability of the first annotator marking correctly as A, and the probability of incorrect marking as 1-A; the probability of the second annotator marking correctly as B, and the probability of incorrect marking as 1-B; the first annotator and the second annotator are two annotators with adjacent numbers. The statistical unit is used to analyze whether the annotation information of the first annotator and the second annotator is consistent for each data to be labeled. If they are consistent, the data to be labeled is added to the dataset. After the analysis of each data to be labeled is completed, the number of data to be labeled in the statistical dataset is counted as N; the number of data to be labeled that each annotator needs to complete is counted as M. The calculation unit is used to calculate two values of A based on formulas (1) and (2), assuming A = B, and the formulas (1) and (2) are: AB + (1-A)*(1-B)=N / M (1) (1-A)*B + A*(1-B) =1- N / M (2) The selection unit is used to choose the larger of the two values of A as the probability of correct answer for the two adjacent annotators.
7. The data annotation device according to claim 5, characterized in that, The annotation module includes: The filtering unit is used to filter out the data with the highest probability of answering correctly and to filter out the labeled data completed by the two annotators corresponding to it, thus completing the initial filtering. The review unit is used to send the initially screened labeled data to reviewers at all levels so that they can conduct multi-level reviews of the labeled information. In each level of review, labeled data that is inconsistent with the review results is deleted, and finally the remaining labeled data is obtained.
8. The data annotation device according to claim 5, characterized in that, The annotation module includes: The training unit is used to select training and test sets from the remaining labeled data using a random grouping algorithm to train the machine learning model; the data to be labeled is taken as input, and the labeling information corresponding to the data to be labeled is taken as output.
9. A data annotation device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data annotation method as described in any one of claims 1 to 4 when executing the computer program.
10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the data annotation method as described in any one of claims 1 to 4.