A data labeling method, device and storage medium
By performing label prediction and uncertainty value calculation on unlabeled data and using thresholds to filter out data to be labeled, the problems of low accuracy and efficiency in the data labeling process are solved, and an efficient data labeling process is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 太保科技有限公司
- Filing Date
- 2022-10-18
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, it is difficult to accurately select data to be labeled during the data labeling process, resulting in a waste of human resources and time costs, and multiple rounds of iterative selection lead to low efficiency.
Label prediction is performed on unlabeled data, its uncertainty value is calculated, and data to be labeled is selected based on a threshold. The data to be labeled is selected by comparing the threshold and the uncertainty value.
It enables more accurate selection of data to be labeled from unlabeled data, saving human resources and time costs, and avoiding the waste of multiple rounds of iterative selection.
Smart Images
Figure CN115577291B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and storage medium for data annotation. Background Technology
[0002] In recent years, artificial intelligence (AI) has developed rapidly, bringing great convenience to people's lives. Only data that has undergone a rigorous labeling process can provide strong data support for AI, driving algorithms to perform machine learning and achieve intelligence. Traditional data labeling mainly relies on manual annotation by data labelers, consuming significant human and time resources. With the development of data processing technology, technicians have discovered that if only a small amount of relatively valuable data—that is, data that is relatively difficult for computers to learn from—can be labeled, and machine learning algorithms can quickly learn from this data, although manual labeling is still necessary, the required human and time resources can be reduced exponentially.
[0003] Existing technologies typically utilize active learning to prioritize data that is difficult for computers to understand. This involves pre-setting the number of unlabeled data points based on experience. However, this method is not precise. If the number of unlabeled data points is too large, more manpower is required for annotation; if the number is too small, it cannot fully label the most difficult data for computers, hindering the implementation of machine learning algorithms. This necessitates re-prioritizing the remaining most difficult data and selecting new points for annotation, requiring multiple iterations and consuming significant time for data labeling.
[0004] Therefore, how to accurately select the data to be labeled during the data labeling process has become an urgent problem to be solved. Summary of the Invention
[0005] In view of this, this application provides a data annotation method, apparatus and storage medium, which aims to accurately select data to be annotated from unannotated data for annotation, thereby saving human resources and time costs in the entire annotation process.
[0006] Firstly, this application provides a data annotation method, the method comprising:
[0007] Perform label prediction on multiple unlabeled data points to obtain the predicted label for each unlabeled data point;
[0008] The uncertainty value of each unlabeled data point is calculated, and the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data point.
[0009] The unlabeled data is filtered based on a threshold and the uncertainty value of each unlabeled data to obtain at least one data to be labeled, wherein the threshold is obtained based on the labeled data;
[0010] Label the selected data to be labeled.
[0011] Optionally, the unlabeled data is filtered based on a threshold and the uncertainty value of each unlabeled data point, including:
[0012] Determine whether the uncertainty value of each unlabeled data point is greater than a threshold;
[0013] Unlabeled data with uncertain values greater than the threshold are selected as data to be labeled.
[0014] Optionally, the threshold is obtained based on labeled data, including:
[0015] Multiple labeled data sets are used as test sets, and a classification model is used to predict the test labels corresponding to each labeled data set.
[0016] Calculate the uncertainty value corresponding to each labeled data in the test set. The uncertainty value is used to measure the reliability of the test label corresponding to each labeled data.
[0017] Based on the distribution of the uncertainty values, an optimal threshold is selected.
[0018] Optionally, based on the distribution of the uncertainty values, an optimal threshold is selected, including:
[0019] Compare each test label with the original label of the corresponding labeled data to see if they are consistent.
[0020] Based on the comparison results between the test label and the original label, a violin plot is drawn using the uncertainty value corresponding to each labeled data in the test set. The violin plot is used to show the distribution and probability density of multiple uncertainty values, and the violin plot contains multiple quartiles.
[0021] Different quartiles in the violin diagram were selected as thresholds;
[0022] Calculate the harmonic mean when the different quartiles are used as thresholds, and the harmonic mean is used to measure the accuracy of using the threshold to filter the data to be labeled;
[0023] The interquartile with the highest harmonic mean was selected as the threshold.
[0024] Optionally, calculating the harmonic mean when the different interquartile ranges are used as thresholds includes:
[0025] Determine whether the test label is consistent with the original label corresponding to each labeled data in the test set. If they are consistent, mark the corresponding labeled data as T; otherwise, mark the corresponding labeled data as F.
[0026] Determine whether the uncertainty value corresponding to each labeled data in the test set is greater than a threshold. If it is greater than the threshold, mark the labeled data as P; if it is less than the threshold, mark the labeled data as N.
[0027] The accuracy rate is defined as the proportion of labeled data marked as TP in the test set to the total labeled data marked as TP or FN.
[0028] The recall rate is the proportion of labeled data marked as TP in the test set to the proportion of labeled data marked as P.
[0029] The harmonic mean was calculated based on the different quartiles as thresholds, using precision and recall.
[0030] Optionally, the uncertainty value of each unlabeled data point among the plurality of unlabeled data is calculated, including:
[0031] The uncertainty value of each unlabeled data in the plurality of unlabeled data is calculated using at least one of the minimum confidence level, the interval method, and the entropy method.
[0032] Secondly, embodiments of this application provide a data annotation apparatus, the apparatus comprising: a prediction label acquisition module, an uncertainty value acquisition module, a data to be labeled acquisition module, and a data annotation module;
[0033] The predicted label acquisition module is used to predict the labels of multiple unlabeled data and obtain the predicted labels corresponding to each unlabeled data.
[0034] An uncertainty value acquisition module is used to calculate the uncertainty value of each unlabeled data in the plurality of unlabeled data, and the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data;
[0035] The unlabeled data acquisition module is used to filter the unlabeled data according to a threshold and the uncertainty value of the predicted label to obtain at least one unlabeled data.
[0036] The data annotation module is used to annotate the selected data to be annotated.
[0037] Optionally, the data to be labeled acquisition module includes: a threshold acquisition submodule and a data to be labeled acquisition submodule;
[0038] The threshold acquisition submodule is used to obtain the threshold based on the distribution of uncertainty values corresponding to the labeled data in the test set;
[0039] The unlabeled data acquisition submodule is used to select unlabeled data with uncertain values greater than a threshold as unlabeled data.
[0040] Optionally, the threshold acquisition submodule is further configured to draw a violin plot based on the comparison results between the test label and the original label of the corresponding labeled data, using the uncertainty value corresponding to each labeled data in the test set, select different quartiles in the violin plot as thresholds, and then select the quartile with the highest harmonic mean as the threshold based on the harmonic mean calculated when the different quartiles are used as thresholds.
[0041] Thirdly, this application provides a computer-readable storage medium storing computer instructions, which, when executed on a data annotation device, perform the steps of the data annotation method described in any one of the first aspects.
[0042] Compared with the prior art, this application has the following beneficial effects:
[0043] This application provides a data annotation method. When executing the method, firstly, label prediction is performed on multiple unlabeled data points to obtain predicted labels for each unlabeled data point. Then, the uncertainty value associated with the predicted label for each unlabeled data point is calculated. Next, the unlabeled data is filtered according to a pre-selected threshold and each of the uncertainty values to obtain at least one data point to be labeled. Finally, the filtered data point to be labeled is labeled. In this way, by selecting a threshold and comparing the threshold with the uncertainty value of the unlabeled data, data to be labeled is selected from the unlabeled data. Therefore, compared to selecting data to be labeled based on experience, this application can more accurately select data to be labeled from the unlabeled data, saving human resources and time costs in the entire data annotation process. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 A flowchart of a method for data annotation provided in the embodiments of this application;
[0046] Figure 2 A flowchart of a method for data annotation provided in the embodiments of this application;
[0047] Figure 3 This is a schematic diagram of a data annotation device provided in an embodiment of this application. Detailed Implementation
[0048] As mentioned earlier, in existing technologies, active learning algorithms are generally used to discover and label relatively valuable data. By using machine learning algorithms to select the data it learns from, higher accuracy in data labeling can be achieved with a small number of training labels.
[0049] In practice, active learning prioritizes the data that is most difficult for computers to understand. Data labelers pre-determine a value 'k' based on experience to represent the number of data points to be labeled. Each time, they select k data points for labeling, thus identifying the relatively valuable data. However, this method has several drawbacks. If the chosen k value is too large, it consumes excessive manpower for labeling; if the k value is too small, too few data points are labeled. The process then repeats, prioritizing the remaining, most difficult-to-understand data and selecting k more data points for labeling. This iterative process consumes significant time. Since selecting data based on experience can lead to wasted human resources and excessive time, a more accurate method for selecting data from unlabeled datasets has become a pressing issue.
[0050] In view of this, embodiments of this application provide a data annotation method. When executing this method, firstly, label prediction is performed on multiple unlabeled data sets to obtain predicted labels for each unlabeled data set; then, the uncertainty value of each unlabeled data set is calculated; next, the unlabeled data is filtered according to a pre-selected threshold and the uncertainty value of each unlabeled data set to obtain at least one data set to be labeled; finally, the filtered data set to be labeled is annotated. In this way, by selecting a threshold and comparing the threshold with the uncertainty value of the unlabeled data to select data to be labeled from the unlabeled data, this application can more accurately select data to be labeled from the unlabeled data for annotation compared to selecting data to be labeled based on experience, saving human resources and time costs in the entire data annotation process.
[0051] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0052] See Figure 1 , Figure 1 A flowchart of a data annotation method provided in this application embodiment, the method comprising:
[0053] S101. Perform label prediction on multiple unlabeled data to obtain the predicted label corresponding to each unlabeled data.
[0054] The unlabeled data can be mainly categorized into data types such as images, videos, audio, and text. This unlabeled data can originate from various sources, including data from multiple databases within enterprise applications, data from online sources such as social media, communication software, mobile communication data, e-commerce online transaction logs, relevant comments from enterprise applications, and user account information, as well as data generated by massive amounts of data from computers or servers, such as application server logs, various sensor data, image and video surveillance data, and QR code and barcode scanning data. Of course, unlabeled data can also be obtained from other sources, without affecting the normal implementation of the embodiments of this application.
[0055] When predicting labels for unlabeled data, several common prediction models can be used to derive the predicted label for each unlabeled data point. For example, a classification model can be used to categorize the data based on information learned from historical data and then label the data with the corresponding predicted label based on the classification results. Alternatively, a clustering model can be used to classify the data into nested intelligent groups based on similar attributes, and then further subdivide each group. An outlier model can also be used to locate abnormal data entries in the dataset, automatically identify the abnormal data, and add labels. Of course, other algorithms or models can also be used to predict labels for unlabeled data, without affecting the normal implementation of the embodiments in this application.
[0056] S102. Calculate the uncertainty value of each unlabeled data in the plurality of unlabeled data, and the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data.
[0057] From a technical perspective, uncertainty value is an indicator that measures the labeling value of unlabeled data based on the amount of information it contains. In other words, the more information unlabeled data contains, the greater the uncertainty value, and the higher the value of the unlabeled data.
[0058] It is understood that, here, the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data. That is, each predicted label corresponding to unlabeled data has an associated uncertainty value, which can facilitate the assessment of the reliability of the computer in predicting the label of the unlabeled data.
[0059] In this embodiment, while performing label prediction on multiple unlabeled data, the uncertainty value of the unlabeled data can be output simultaneously with the corresponding predicted label. The order of output is not limited.
[0060] In practice, the uncertainty value can be calculated using the minimum confidence level, the interval method, or the entropy method in uncertainty sampling. The minimum confidence level method uses a threshold value derived from pre-provided conditions to represent the lowest reliability. When calculating the uncertainty value, the probability that each unlabeled data point does not correspond to its corresponding predicted label is calculated by measuring the confidence level of the predicted label's correspondence with its unlabeled data. The interval method primarily considers the degree of difference between the two predicted labels corresponding to each unlabeled data point to calculate the uncertainty value of the unlabeled data. The entropy calculated in the entropy method is a measure of the uncertainty value. The greater the consistency between the predicted label and the unlabeled data, the smaller the entropy and the smaller the uncertainty value; conversely, the smaller the consistency, the larger the entropy and the larger the uncertainty value. S103: The unlabeled data is filtered according to the threshold and the uncertainty value of each unlabeled data point to obtain at least one data point to be labeled. The threshold value is obtained based on the labeled data.
[0061] The threshold is used to define the minimum uncertainty value for selecting unlabeled data from the unlabeled data to be labeled. Unlabeled data with uncertainty values greater than the threshold can be selected as labeled data by determining whether the uncertainty value of each unlabeled data is greater than the threshold.
[0062] The threshold can be determined by predicting the test labels corresponding to each labeled data point in the test set using a classification model, and selecting it based on the distribution of uncertainty values associated with the test labels corresponding to each labeled data point. Alternatively, it can be automatically generated in other ways, such as by forming multiple training and test sets and training a neural network model using a similar algorithm, with the threshold automatically generated by the neural network model. Neither of these methods affects the normal implementation of the embodiments of this application.
[0063] The data to be labeled refers to the data selected from the unlabeled data that will be labeled, and it can be represented as one or more datasets.
[0064] S104. Label the selected data to be labeled.
[0065] Data annotation mainly refers to the classification, organization, editing, error correction, labeling and annotation of data of data types such as images, videos, audio and text, adding labels to the data to be annotated, and generating machine-readable data codes that meet the requirements of machine learning training.
[0066] Here, the annotation of the data to be annotated is mainly done manually by data annotators. Of course, using other methods that can accurately annotate the data will not affect the normal implementation of this application's embodiments.
[0067] This application provides a data annotation method. When executing this method, firstly, label prediction is performed on multiple unlabeled data sets to obtain predicted labels for each unlabeled data set; then, the uncertainty value of each unlabeled data set is calculated; next, the unlabeled data is filtered based on a pre-selected threshold and the uncertainty value of each unlabeled data set to obtain at least one data set to be labeled; finally, the filtered data set to be labeled is annotated. In this way, by selecting a threshold, the uncertainty of the unlabeled data is used to select the data to be labeled. Therefore, compared to selecting a certain amount of data to be labeled based on experience, this application can more accurately select data to be labeled from unlabeled data, allowing for reasonable allocation of human resources for data annotation and eliminating the need for multiple rounds of iterative data selection and annotation, thus saving time costs.
[0068] In the embodiments of this application, the above Figure 1 There are multiple possible implementations for obtaining the threshold in step S103, which are illustrated below with examples in specific scenarios. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.
[0069] See Figure 2 The figure is a flowchart of a data annotation method provided in an embodiment of this application. The data annotation method includes:
[0070] S201. Using multiple labeled data sets as a test set, a classification model is used to predict the test label corresponding to each labeled data set.
[0071] Here, the test label refers to the computer-predicted label for each labeled data point in the test set, used for comparison with the original label of the corresponding labeled data.
[0072] Of course, other models can also be used to predict the test labels of the test set, such as clustering models, outlier models, etc., which are similar to the methods used to obtain predicted labels for unlabeled data, and will not be elaborated here.
[0073] S202. Calculate the uncertainty value corresponding to each labeled data in the test set. The uncertainty value can be calculated using methods in uncertainty sampling, such as the interval method, to determine the uncertainty value corresponding to each predicted label. Of course, uncertainty values calculated using other methods are also within the scope of protection of this application.
[0074] S203. Compare whether each test label is consistent with the original label of the corresponding labeled data;
[0075] In this process, each test label can be compared with the original label of the corresponding labeled data to determine whether they are consistent. Then, the uncertainty value corresponding to each labeled data in the test set can be calculated. That is to say, the order of steps S202 and S203 is not limited in this embodiment.
[0076] S204. Based on the comparison results between the test label and the original label, a violin plot is drawn using the uncertainty value corresponding to each labeled data in the test set. The violin plot is used to show the distribution and probability density of multiple uncertainty values. The violin plot contains multiple quartiles.
[0077] A violin plot is a method for plotting continuous data. It can be used as a combination of a box plot and a kernel density plot. A box plot shows the location of quantiles, while a kernel density plot shows the density at any location. A violin plot can show the distribution and probability density of multiple sets of data, revealing which locations have a high concentration of data points.
[0078] Specifically, drawing a violin plot based on the comparison results between the test label and the original label, using the uncertainty value corresponding to each labeled data in the test set, means drawing a violin plot based on the uncertainty value of the labeled data whose test label and the original label are consistent, drawing another violin plot based on the uncertainty value of the labeled data whose test label and the original label are inconsistent, and combining the two violin plots to form the violin plot of this application embodiment.
[0079] S205. Select one of the quarter points in the violin diagram as the threshold.
[0080] In this context, quartiles refer to the division of all values on the vertical axis of a violin diagram into four equal parts from smallest to largest. The values located at the three dividing points are called quartiles, which mainly include the upper quartile, the middle quartile, and the lower quartile.
[0081] The violin plot here may contain six quartiles: the upper quartile, the middle quartile, and the lower quartile in the violin plot drawn based on the uncertainty value of the labeled data that is consistent with the test label and the original label; and the upper quartile, the middle quartile, and the lower quartile in the violin plot drawn based on the uncertainty value of the labeled data that is inconsistent with the test label and the original label.
[0082] Of course, other quantiles, such as ten percentiles and percentages, can also be selected here, and they are also within the protection scope of this application's embodiments.
[0083] S206. Calculate the harmonic mean when the quartile is used as a threshold. The harmonic mean is used to measure the accuracy of using the threshold to filter the data to be labeled.
[0084] The harmonic mean is an indicator used to measure the accuracy of a binary classification model. In the embodiments of this application, it can be used to measure the accuracy of filtering data to be labeled using a threshold. The binary classification model is also known as a two-value classification model, which is a classification model in which the prediction result can be formally expressed as 0 or 1.
[0085] The harmonic mean can be calculated using the following method:
[0086] Determine whether the test label is consistent with the corresponding original label. If they are consistent, mark the corresponding labeled data as T. If they are inconsistent, mark the corresponding labeled data as F.
[0087] Determine whether the uncertainty value corresponding to each labeled data in the test set is greater than the threshold. If it is greater than the threshold, mark the labeled data as P; if it is less than the threshold, mark the labeled data as N.
[0088] Of course, this is merely an example. Different methods are used to calculate uncertainty values, and the comparison methods for determining the data to be labeled also differ. For instance, using the minimum confidence method, if the uncertainty value corresponding to the labeled data is greater than the threshold, the corresponding labeled data is marked as P; if it is less than the threshold, the labeled data is marked as N. Using the interval method, if the uncertainty value corresponding to the labeled data is greater than the threshold, the corresponding labeled data is marked as P; if it is less than the threshold, the labeled data is marked as N. Using the entropy method, if the uncertainty value corresponding to the labeled data is greater than the threshold, the corresponding labeled data is marked as N; if it is less than the threshold, the labeled data is marked as P.
[0089] Of course, if other methods are used to calculate the uncertainty value, the comparison method for determining the data to be labeled using the threshold will also be different, which will not be elaborated here.
[0090] The precision rate is defined as the proportion of labeled data marked as TP in the test set to the total labeled data marked as TP or FN.
[0091] The recall rate is the proportion of labeled data marked as TP in the test set to the proportion of labeled data marked as P.
[0092] The harmonic mean was calculated based on the different quartiles as thresholds, using precision and recall.
[0093] S207. Repeat steps S204 and S205 until all quartiles have been selected.
[0094] Of course, you can also pre-set other quantiles to be selected, or discard some quantiles with relatively low value to save time in selecting the threshold.
[0095] S208. Select the quartile with the highest harmonic mean as the threshold.
[0096] When the quartiles are selected as the threshold, the accuracy of selecting the data to be labeled from the unlabeled data is higher than that of selecting other quartiles as the threshold.
[0097] This application embodiment calculates the harmonic mean of each quantile as a threshold, and selects the quartile with the highest harmonic mean as the threshold, which makes the selection of data to be labeled more accurate and can save manpower and time resources in the labeling process.
[0098] The above are some specific implementations of the methods provided in the embodiments of this application. Based on this, this application also provides corresponding apparatus. The apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.
[0099] See Figure 3 The diagram shows the structure of the data labeling device 300, which includes a prediction label module 310, an uncertainty value acquisition module 320, a data to be labeled acquisition module 330, and a data labeling module 340.
[0100] The predicted label acquisition module 310 is used to predict the labels of multiple unlabeled data and obtain the predicted labels corresponding to each unlabeled data.
[0101] The uncertainty value acquisition module 320 is used to calculate the uncertainty value of each unlabeled data in the plurality of unlabeled data, and the uncertainty value is used to measure the reliability of the prediction label corresponding to each unlabeled data.
[0102] The uncertainty value acquisition module 320 can also be used to calculate the uncertainty value of each unlabeled data in the plurality of unlabeled data using at least one of the minimum confidence level, the interval method, and the entropy method.
[0103] The unlabeled data acquisition module 330 is used to filter the unlabeled data according to a threshold and the uncertainty value of the predicted label to obtain at least one unlabeled data.
[0104] The data acquisition module 330 can include a threshold acquisition submodule and a data acquisition submodule.
[0105] The threshold acquisition submodule is used to obtain the threshold based on the distribution of uncertainty values corresponding to the labeled data in the test set;
[0106] The unlabeled data acquisition submodule is used to select unlabeled data with uncertain values greater than a threshold as unlabeled data.
[0107] Of course, the threshold acquisition submodule can also be used to draw a violin plot based on the uncertainty value corresponding to each labeled data in the test set, select different quartiles in the violin plot as thresholds, and then select the quartile with the highest harmonic mean as the threshold based on the calculated harmonic mean of the different quartiles as thresholds.
[0108] The data annotation module 340 is used to annotate the selected data to be annotated.
[0109] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0110] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0111] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.
Claims
1. A data annotation method, characterized in that, The method is applied to machine learning model training, and the method includes: Label prediction is performed on multiple unlabeled data to obtain the predicted label corresponding to each unlabeled data; wherein, the unlabeled data includes image, video, audio or text data; The uncertainty value of each unlabeled data point is calculated, and the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data point. The unlabeled data is filtered based on a threshold and the uncertainty value of each unlabeled data to obtain at least one data to be labeled, wherein the threshold is obtained based on the labeled data; The selected data to be labeled is then labeled for use in training machine learning models; The step of filtering the unlabeled data based on a threshold and the uncertainty value of each unlabeled data point includes: Determine whether the uncertainty value of each unlabeled data point is greater than a threshold; Unlabeled data with uncertainty values greater than a threshold are selected as data to be labeled; The threshold is obtained based on labeled data, including: Multiple labeled data sets are used as test sets, and a classification model is used to predict the test labels corresponding to each labeled data set. Calculate the uncertainty value corresponding to each labeled data in the test set. The uncertainty value is used to measure the reliability of the test label corresponding to each labeled data. Compare each test label with the original label of the corresponding labeled data to see if they are consistent. Based on the comparison results between the test label and the original label, a violin plot is drawn using the uncertainty value corresponding to each labeled data in the test set. The violin plot is used to show the distribution and probability density of multiple uncertainty values, and the violin plot contains multiple quartiles. Different quartiles in the violin diagram were selected as thresholds; Calculate the harmonic mean when the different quartiles are used as thresholds, and the harmonic mean is used to measure the accuracy of using the threshold to filter the data to be labeled; The interquartile with the highest harmonic mean was selected as the threshold.
2. The method according to claim 1, characterized in that, Calculate the uncertainty value of each unlabeled data point among the plurality of unlabeled data points, including: The uncertainty value of each unlabeled data in the plurality of unlabeled data is calculated using at least one of the minimum confidence level, the interval method, and the entropy method.
3. A data annotation device, characterized in that, The device is used for training machine learning models, and the device includes: a prediction label acquisition module, an uncertainty value acquisition module, a data to be labeled acquisition module, and a data labeling module; The predicted label acquisition module is used to predict labels for multiple unlabeled data to obtain the predicted labels corresponding to each unlabeled data; wherein, the unlabeled data includes image, video, audio or text data; An uncertainty value acquisition module is used to calculate the uncertainty value of each unlabeled data in the plurality of unlabeled data, and the uncertainty value is used to measure the reliability of the predicted label corresponding to each unlabeled data; The unlabeled data acquisition module is used to filter the unlabeled data according to a threshold and the uncertainty value of the predicted label to obtain at least one unlabeled data. The threshold is obtained based on the labeled data and includes: using multiple labeled data as a test set, using a classification model to predict the test label corresponding to each labeled data; calculating the uncertainty value corresponding to each labeled data in the test set, the uncertainty value being used to measure the reliability of the test label corresponding to each labeled data; and comparing whether each test label is consistent with the original label of the corresponding labeled data. The data annotation module is used to annotate the selected data for use in training machine learning models. The module for acquiring data to be labeled includes: a threshold acquisition submodule and a data acquisition submodule to be labeled; The threshold acquisition submodule is used to acquire the threshold based on the distribution of uncertainty values corresponding to the labeled data in the test set. The submodule for acquiring unlabeled data is used to select unlabeled data with an uncertainty value greater than a threshold as data to be labeled. The threshold acquisition submodule is further configured to draw a violin plot based on the comparison results between the test label and the original label, using the uncertainty value corresponding to each labeled data in the test set, select different quartiles in the violin plot as thresholds, and then select the quartile with the highest harmonic mean as the threshold based on the calculated harmonic mean of the different quartiles as thresholds.
4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when executed on the data annotation device, perform the steps of the data annotation method as described in any one of claims 1-2.